I run a review pass on top of AI-generated findings — code review comments, "this is a bug" claims, that kind of thing — where a second pass of agents tries to knock each finding down before it's shown to me. The idea is simple: spawn a few independent skeptics, each told to argue the finding is wrong, and only keep findings that survive a majority vote. It catches the plausible-but-wrong stuff that a single confident agent produces constantly — the finding that reads perfectly, cites real line numbers, and describes a bug that doesn't actually happen under any input.
It worked, for a while. Going from zero verification to three-vote majority verification killed a huge chunk of false positives — the kind where the "bug" is a scenario that can't actually occur, or a misread of what the code does. That was the easy win. Then I tried to push further: five verifiers, seven verifiers, assuming more votes would keep buying more precision. It didn't. Past three-ish, adding another identical verifier barely moved the false-positive rate at all, it just cost more tokens per finding.
Why more of the same vote doesn't help
The verifiers weren't independent in any way that mattered. They were the same prompt — "try to refute this claim, default to refuted if uncertain" — run three, five, seven times. Same framing, same blind spots, same tendency to accept a finding that sounds rigorous because it uses the right vocabulary (specific line numbers, a named function, a plausible-sounding failure mode) even when the underlying claim is wrong. If verifier #1 gets fooled by a finding that name-drops the right symbols, verifiers #2 through #7 get fooled by the exact same thing, for the exact same reason. You're not sampling independent opinions. You're re-rolling the same biased die and calling the agreement "confidence."
