TL;DR. To catch a drop from a 0.90 pass rate to 0.85 at 80% power (one-sided, alpha 0.05), you need about 253 examples. A 50-example set has roughly 35% power, so it misses that regression about two times in three. The move that matters is not "collect more data" as a slogan. Size the set to the effect you actually care about, report a confidence interval on each run instead of a bare point delta, and prefer per-criterion binary labels over vague graded scores.
The two-point win that wasn't
A few months ago I sat in a review where a team was pleased with itself. A new prompt had moved their eval pass rate from 87.5 percent to 90 percent, and someone had already written "prompt v3: +2pp" in the changelog. The eval set had forty examples.
Here is what those points were made of. On forty examples every result is a multiple of 2.5 points, so the pass rate can only land on 35 out of 40, or 36, or 37, with nothing in between. The move from 87.5 to 90 was 35 correct becoming 36 correct. One example. A single test case that used to fail now passed, and it could flip back next week when the decoding lands differently.
I asked the obvious question. If we reran the old prompt three more times, would it always score 35 out of 40? Nobody knew, because nobody had rerun it. The honest summary of that meeting is that we had watched one example change its mind, and we had written it into the changelog as progress.
