I re-ran the same measurement, and the bottleneck had moved
A few months later I re-ran the same measurement, and the breakdown had flipped. Generation was 10% of the total. The remaining 90% was queueing and distance.
The lever that had worked best last time was now beside the point. The bottleneck moves — right answers have a shelf life.
So I tried running inference closer to the user (edge inference) to eliminate the round trip, and this time the whole thing got 3x slower. Local optimization loses on the whole.
Last time I wrote that the cause of the slowness was the model's generation — more precisely, that the model was "thinking" before it answered. I turned the reasoning off (reasoning=none), and generation got dramatically lighter all at once. At the time, that was enough.
