Outcome grading is the obvious choice and mostly the right one: did the end state match what was asked? It is cheap, objective and hard to game. It is also blind to a set of failures that will hurt you in production, and every one of them is invisible in a green test run.
Outcome grading, and what it cannot see
Run the agent, inspect the final state — the file’s contents, the database row, the returned answer — and compare against an expectation. Where the environment can verify the outcome by execution (do the tests pass? does the row exist?) this is the strongest signal available, and nothing below argues for dropping it.
Five failures it passes anyway:
Right answer, wrong reason. The agent guessed, or answered from parametric knowledge without consulting the tool that was the point of the test. It will get the next, unseen case wrong, and your eval will not have predicted that.
