Last month I audited the statistics inside Mia, my ad-testing agent, and found a 20x error that had never broken anything. No exception, no failed assertion, nothing in the logs. The textbook shortcut said each arm of an A/B test needed 1,111 users. The exact formula, run at the 1.9 percent click-through rate ads actually get, said 22,278. Every test Mia had sized was running on a twentieth of the data it needed, and every one of them still announced a winner. I run AI systems in several industries, and the failures that cost the most all have this shape: nothing breaks, and the answer is wrong.
Where the shortcut came from
The shortcut is Lehr's equation: sixteen over the effect size squared. It is in the textbooks because it is easy, and it is easy because it was derived where the math is friendliest, near coin-flip rates, where variance peaks and everything is symmetric. Ads do not live there. Ads live at 1.9 percent, where the effects worth catching are tiny in absolute terms and the samples you need to see them are huge. The shortcut knows none of this. It returns 1,111 and moves on.
The part that took me longest to accept is that the formula is not wrong. It works fine in the world it came from. The bug was mine. I carried it into a different world without re-deriving it.
