Last week I was running a research task. At the end, I asked another AI to check whether the output met the requirements. It came back: passed.
I didn't look any further. After all, this was a "double-check" — it should be more reliable than just me reviewing it alone.
Then I opened that output for an unrelated reason, and found two obvious logical breaks. The kind you'd notice just from reading straight through.
Not an occasional slip. The verification itself had never really happened.
I looked into it, and it turns out there's a batch of researchers currently doing exactly this: quantifying how reliable AI actually is at judging whether a task is complete. One evaluation called AgentProp-Bench uses the kappa coefficient — the standard measure of inter-rater agreement — and found that substring-based AI judgment methods score 0.049. That's close to random.
