A pattern I've seen on more than one team: weekly eval run finishes, someone sorts the leaderboard, and the worst-performing prompt variant or model checkpoint gets flagged for attention. Someone makes a change, a tweak to the system prompt, a different few-shot example, sometimes just a rewording of one instruction. Next week's eval run shows that variant improved. The team credits the fix.
Sometimes the fix is real. Often, some or all of that improvement would have happened anyway, with no change at all, just because of how "worst performer" gets selected.
The mechanism
If your eval set is small (a few hundred examples, sometimes less) or your judge scoring is noisy (LLM-judge variance, human rater disagreement, both), then any variant's measured score on a given week is its true underlying quality plus some random noise. Most weeks, that noise roughly cancels out. But the variant you pick out as "worst this week" wasn't selected at random. It was selected because it had a low score, and a low score is disproportionately likely to belong to a variant that both has middling true quality and got unlucky that week.
Next time you measure it, the "got unlucky" part isn't guaranteed to repeat. On average, it won't. So the score tends to drift back up toward that variant's actual mean, regardless of whether anyone touched the prompt. This is regression to the mean, and it's one of the oldest documented statistical phenomena (Galton described it in the 1880s studying the heights of parents and children, well before anyone was eval-ing a prompt), and it shows up anywhere you select on an extreme observation and then re-measure.
