Reading through the awesome-evals list on GitHub yesterday, I realized something I've been circling for months: most of us are benchmarking wrong.
Not "wrong" as in bad numbers. Wrong as in measuring the wrong thing.
The list is a curated library of papers, tools, and benchmarks for evaluating AI agents — no fluff, no vendor pitches. What struck me is how many evals test whether a model can do something in isolation, and how few test whether it will do the right thing when the environment is messy, the instructions are ambiguous, and the tool call fails three times in a row.
That's the gap. Lab evals measure capability. Production evals measure reliability.
I've been running a small self-hosted agent stack for a few months now, and the bugs that actually bite me are never "the model couldn't answer this question." They're "the model tried to call a tool with a malformed argument because the previous step returned something unexpected." Or "the model got stuck in a loop because it didn't recognize the error message."
