I Fabricated a Claim About LLM Judges. Then I Ran the Apology Experiment.

Where this fits: A series aside, not a numbered Part. It is first an apology for a fabricated claim under Part 3; the experiment below is how I made amends. Numbers: scripts/results-v2/*_summary.json (+ matching .jsonl). One nesting note: the original comment was wrong, and the first version of this apology was also wrong about one scenario (DS4). The figures here are the corrected ones.

This piece is an apology.

In a comment under Part 3, I wrote:

"Part 3 found that judges fail on directional failures — outputs that were semantically reversed (delete → keep, stop → continue) but structurally pristine."

That claim was false. (Public retraction on the same thread.)

I Fabricated a Claim About LLM Judges. Then I Ran the Apology Experiment.

Related reading

Five Adversarial Reviews Told Me My Study Was Measuring a Fiction

How I Caught My LLM Fabricating Its Own Evidence

I Built an Adversarial Eval Framework and Attacked 5 LLMs — Every Single One…

Five Comments That Redesigned My LLM Verification Pipeline

A Better LLM Judge? The Rubric Made My Small Model Worse

I Planned 10 LLM Evaluation Experiments And Only Ran 1. It Was Enough.