Passing the eval is not solving the task: a green agent eval certifies nothing if the agent can write the files the grader reads, or reach the reference answer. This 60-line static probe read one harness spec, flagged 3 contamination points (2 write-read, 1 reference leak), and exited 1 without running the agent.
I keep seeing the same screenshot in agent-eval threads: a wall of green checkmarks, "98% pass rate," ship it. Then the same agent face-plants in production on a task that looked identical to a passing eval case.
The usual explanation is "the eval set is too easy" or "distribution shift." Sometimes. But there's a quieter failure that nobody screenshots, because it never shows up as red: the harness graded a number the agent itself wrote.
Here's the claim I'll defend with a runnable tool: passing the eval is not the same as solving the task. A green run only certifies the agent if two things hold. The channel the agent can write into is disjoint from the channel the grader reads from. And the reference answer is not sitting somewhere the agent can open before grading. Break either one and your "98%" is, by construction, undecidable. The agent could have aced it. It could also have written {"passed": true} to the file the grader trusts. The score can't tell you which.
