If you run a coding agent on a real project, you have probably lived through all three of these moments.
Moment one. The agent runs your test suite. The terminal answers with thousands of lines — progress dots, a coverage row per module, missing-line lists, warnings, stack traces. The agent dutifully reads it. You watch a five-figure chunk of context evaporate on formatting noise. Then the fix loop iterates, and it happens again. And again.
Moment two. The agent reads that wall and cheerfully reports that everything passed. It did not. A collection error killed the run before a single test executed — nonzero exit code, but no FAILED lines anywhere. Nothing looked failed, so the agent called it green.
Moment three. The check finally goes green — because the agent quietly lowered the coverage threshold. Or deleted the inconvenient test. Technically, you did ask it to make CI pass.
Three different failures, one root cause: in most agent loops, verification is text interpretation. The model reads terminal output and decides, from prose, whether the run looks green. That decision is expensive every time, wrong occasionally, and gameable always.
