TL;DR— Benchmark scores report central tendency over a fixed, static distribution of test items, but production reliability is governed by tail behavior on a shifting distribution of real inputs. A model can post a great average and still fail unpredictably on the exact slice of traffic your product depends on. Teams that only track leaderboard deltas are optimizing the wrong statistic.
A benchmark score is a mean. That sentence sounds obvious, but almost nobody treats it that way. Teams read 84.3 on MMLU or 91% pass rate as a proxy for how good a model is, full stop. It isn't. It's the average outcome over a fixed, curated distribution of test items, scored under a fixed protocol. Production is not a fixed distribution, and your users are not sampling uniformly from a benchmark's test set. The gap between those two facts is where most eval-driven decisions quietly go wrong.
The scalar illusion
Reducing model behavior to one number is a compression trick, and every compression trick throws away information. When you collapse thousands of task-level outcomes into a single accuracy figure, you are explicitly discarding the shape of the error distribution: which items failed, how badly, how consistently, and whether failures cluster in a way that maps to something a real user would hit. Two models can post identical aggregate scores while having completely different failure profiles. One fails randomly and rarely. The other fails reliably on a specific category of input— long documents, ambiguous negation, multi-hop arithmetic, non-English names— and never on anything else. The leaderboard cannot tell you which one you're looking at. Your incident channel can.
