If you're building anything with an LLM judge in the loop, this is the failure mode that will get you, and you won't see it happen. I didn't, until I went looking for the opposite.
The story, in the order it happened.
The thing I was building
I wanted to measure something specific: can an AI coding agent navigate a real codebase, not just read one file, but answer "what depends on this model, everywhere, before I change it." That's the question a maintainer answers in their head before a risky refactor, and it's the one an agent tends to get confidently wrong.
So I built a benchmark. Pick the hub model of a real app, the Inbox in Chatwoot, and ask the agent to find every dependent before a teardown change. Run it two ways: a plain agent that greps and infers, and the same agent handed a structural map of the codebase it can query. Same model, same prompt, same pinned commit. Measure the difference.
