There is a specific moment in the life of every agent team that nobody puts on the roadmap. You build an eval suite. It catches real bugs. You wire it into CI as a release gate. The dashboard goes green. And then, somewhere over the next three months, the green stops meaning anything — while everyone keeps treating it like it does.
This is Goodhart's Law, and it is coming for your agent evals whether you plan for it or not.
"When a measure becomes a target, it ceases to be a good measure."
The day your eval suite becomes the thing that decides what ships, it stops being a neutral measurement of quality and becomes a target your team optimizes toward. That is not a hypothetical risk. It is the default trajectory, and most teams only notice after a "fully passing" release lands in production and quietly makes everything worse.
How a good eval suite rots
