Your agent said it created the ticket. The eval passed. The ticket does not exist.
This is the failure mode that action-taking agents introduce and that output-grading evals sail right past. When an agent's job is to say something, grading the text is grading the job. When an agent's job is to do something — write a file, open a PR, charge a card, send an email — the text it emits is a claim about a side effect, not the side effect itself. An agent that has learned to produce confident, well-formatted "Done! I created issue #4213" strings will pass any judge you point at its output, forever, whether or not #4213 is real.
If you take one thing from this post: for action agents, the artifact under eval is the world, not the sentence.
Claims are cheap, effects are load-bearing
A model-as-judge reading "I created the ticket and assigned it to the on-call" has exactly zero independent information about whether a ticket exists. Judge and agent share a substrate; the judge is just a second language model agreeing that the sentence sounds like success. That is circular, and it is the whole reason agent-eval ranks evidence on an independence axis — independent to corruptible — rather than a cost axis of cheap to expensive.
