Here's a question almost no eval pipeline can answer: if you asked your LLM judge to score the exact same response five times, would you get the same number back?
Most teams never check. They wire up an LLM-as-judge, pick a threshold that feels right ("block anything under 7"), and ship it as a merge gate. Then they spend months wondering why the same PR is green on one run and red on the next. The judge wasn't wrong. It was unstable, and nobody measured the stability before trusting the number.
I built maf-evals — a three-tier agent evaluation reference on Microsoft Agent Framework and .NET 8 — partly to force this discipline into the open. The rule that fell out of it: measure the judge before you trust it, and measure self-consistency before you even think about human agreement. Every number below came from actually running the calibration command, not from reasoning about it.
The two questions, in the only order that works
When you calibrate a judge, there are two questions:
