An agentic benchmark does not compare a string to an answer key. It puts a system into an environment, lets it act, and then inspects the environment. That is a much better measurement of whether something worked — and it makes the resulting number depend on a great deal more than the model.
What makes an agentic benchmark different
On MMLU or HumanEval the system produces one artefact and grading is a function of that artefact. In an agentic benchmark the system takes a sequence of actions, each changing the state it observes next, and grading is a function of the final state. Three things follow.
Many correct trajectories exist. There is usually more than one route to the goal. Grading the outcome rather than the path is the only tractable choice, and it means the benchmark cannot distinguish an efficient solution from a lucky one — the trade-off is discussed in trajectory versus outcome grading.
Errors compound. A twenty-step task where each step succeeds independently 95 per cent of the time completes 36 per cent of the time. Agentic scores are low for arithmetic reasons before any capability question arises.
