A validator that reports a verdict without reporting its coverage is asserting something it did not measure. I found that in four systems before I admitted it was one problem.
Originally published at ai.bedvibe.studio.
Most validation tooling has two states: it passed, or it failed. Everything that was not actually evaluated has to be forced into one of them — and it is wrong in both directions.
I did not work that out from first principles. I worked it out by shipping the same defect four times.
One: a run that learned nothing, reported as healthy
In trainproof, a training run whose loss was exactly 0.0 on every step returned PASS.
I pointed a 22-agent review at my own competition endgame expecting notes about the code. It came back with three errors in my…
Last week's eval on a truncated JSON bug showed us something uncomfortable: DebugAI can diagnose a...
Across one of our agents, the large majority of genuine defects - eight out of nine, when I went back...
We run a small system that audits itself every few hours. Each cycle the agent produces a verdict...
A verifier holding its own copy of the rule it checks, a pinned PASS that only asserted the assertion count, and an estimator…
My build fails when two copies of a fact disagree — but a fact wrong in every copy at once sails through, because agreement isn't…
