My coding agent ends almost every task the same way. "Done! All tests pass." For months I just believed it, and for months I kept finding the lie a day later. The test that never ran. The env var that never got set. The endpoint answering 500.
Turns out this is measured. A paper from June (arXiv 2606.09863) looked at agent runs that graded themselves and found that among the failing ones, 75.8% still claimed success. The same paper tried using LLM judges to catch it. Best AUROC across 5 judges and 5 prompt strategies was 0.65, worst was 0.54. A coin flip, because judges read the confident closing tone instead of the actual state of the machine.
A test runner detects a failing suite at 1.0. So I built a test runner wearing a Stop hook.
The receipt
nuhuh treats the agent's final message as a list of hypotheses. It extracts every claim ("all tests pass", "created src/x.ts", "endpoint works", "set DATABASE_URL") and re-runs reality, fresh. Whole suite in a clean process reading real exit codes, files checked on disk, localhost actually called. Then it prints a receipt it wrote, not one the agent dictated.
