You wrote a new eval. It caught the failure you just saw in production. You shipped it as a gate. Congratulations — you now have a piece of untested code sitting on the hot path of every agent run, deciding what ships and what gets blocked.
We treat agent evals as if writing them is the hard part. It isn't. The hard part is knowing your eval actually discriminates: that it goes red on the bad traces and green on the good ones, and not the other way around. An eval you haven't run against known-labeled traces is a coin flip with a dashboard.
The failure nobody names: the false-negative gate
Here's the quiet disaster. You add a rule to catch hallucinated file paths. It has a regex bug. It matches nothing. Every run passes. Your dashboard is green. You feel safe. Three weeks later a customer finds the exact failure your "gate" was supposed to block, and you discover the rule never fired once.
A green eval is not evidence of a healthy agent. It's evidence that either the agent is healthy or your eval is broken — and you have no way to tell those apart unless you've fed it a trace you already know is bad and watched it go red.
