This is a companion to the PlannerCritic series. Article 2 was about a specific critic bug. This one is about the design principle I extracted from fixing it — and the measurement that proved it holds.
I measured my LLM critic on identical input across five trials. It returned a different verdict every single time. label_flip_rate = 1.0. It also never let a defective plan through. underclaim_approvals = 0. Both are true. The frozenset is why.
The Uncomfortable Measurement
In v0.2.1 I added a test I'd been avoiding: send the same boundary-case plans through the real critic model five times and measure what changes. The live-critic boundary evaluator (#218).
The numbers:
