A while back I was building a support assistant and hit a simple-sounding question: is this new version of the prompt actually better than the old one? So I did the obvious thing. I wrote thirty test cases, ran both prompts, saw the new one score a little higher, and shipped it.
I felt great about it for half a day, until the support queue filled with complaints and I was rolling the change back from a Slack thread that evening.
The bump in the score was never real. The test was far too small to tell a tiny genuine improvement apart from luck, so the number was noise the whole time. That one lesson is behind almost everything below. This is the guide I swear by now, in plain language, and I hope it saves you the same evening.
First, figure out how small a change your test can even see
A small test can only catch large differences. If the real improvement is small, thirty examples cannot tell it apart from the random wobble of which examples you happened to pick. Run it again and it often flips.
