A team I compared notes with recently runs continuous evals on production traffic: an LLM judge scores a sample of traces, the scores land on a dashboard, and a monitor pages when the hourly pass rate drops below 88 percent. Their baseline is around 92. The monitor paged on four separate days last week. Four investigations, four shrugs, four "it recovered on its own."
Nobody chose a false-alarm rate for that monitor. But it has one, and it is not small. With roughly 150 judge scores an hour, a fresh window each check, and a true pass rate of 0.92, the chance that at least one hourly check dips below 0.88 at some point in a day is 53 percent. That makes the expected number of alert days in a week 3.7, and the single most likely weekly count exactly four. Their monitor was not detecting regressions. It was sampling noise on a schedule, and the on-call rotation was the readout.
An alert threshold on an eval score is a hypothesis test. Every tool that ships eval monitoring runs that test for you. What none of the configuration surfaces I read this month asks you for is the one number the test depends on: how many scores are in the window.
The test you are actually running
The setup is ordinary binomial arithmetic, which is what makes it checkable. Each check looks at a fresh window of n judge verdicts. The true pass rate is p. The monitor fires when the observed rate falls below a threshold t. Then:
