Validators Judge. They Don't Help.

Originally published at devopsdiary.blog. Post F-AID3 in the "Governing AI in the Enterprise" series.

A validator that fixes the thing it just flagged has quit its job. It stopped grading and started doing the homework, in the same motion, and now you can't tell which part of the result you're supposed to trust.

I keep running into this in the tooling everyone's shipping right now. The AI-dev vendors have converged on a lot of good ideas over the past year. Most of them I want to steal. One of them I'll fight about, because it quietly breaks the one property that made the whole setup governable in the first place.

Let me take the good ones first. I'm not interested in being the guy who only shows up to say no.

Three ideas worth stealing

Validators Judge. They Don't Help.

Related reading

Validate Your AI Judge With Negative Controls

A RAG evaluator that admits what it can't judge

Reliable, and still wrong

The AI judge that called a half-finished audit 'exhaustive'

The AI writes the tests. It doesn't get to grade them.

The Direction of Blame