Originally published on hexisteme notes.
I run a small fleet of coding agents on one machine. Every thread ends up in a log, and a measurement pipeline reads those logs into a database, attributing each turn to the model that produced it. After a few thousand threads I had the table people keep asking for: seven model-versions, five behavioural metrics, real workload rather than a benchmark.
Then I printed one cross-tab I had been skipping, and most of that table stopped meaning what I thought it meant.
The cross-tab was role × model. In this fleet a model runs in one of two roles. It is either the long interactive main thread I drive by hand, or a short one-shot sub-agent that a main thread spawns, runs once, and discards. Same model. Same weights. Two completely different jobs.
Role turned out to move the numbers by up to 135x, and the role mix is wildly different for each model. Those two facts together are enough to make a pooled comparison manufacture a large gap that exists in neither stratum.
