We talk about agents drifting. We almost never talk about the thing we measure them against drifting. But your golden dataset — the fixtures, expected outputs, and "known good" traces your evals grade against — is code that ships to production and then never gets a code review again. It rots. And when your oracle rots, a green dashboard stops meaning your agent is correct. It means your agent still agrees with a snapshot of the world you took eight months ago.
This is the failure mode senior teams keep rediscovering the hard way: the agent is fine, the harness is fine, and the test oracle is the thing that's wrong. Nobody re-validates it, because passing tests feel like the end of the story instead of a claim that also decays.
Why oracles rot
A golden dataset encodes assumptions about the world at the moment you captured it. A few months later:
The API you called changed its response shape, so your "expected output" is now describing an endpoint that no longer exists.
