Originally published on hexisteme notes.
We built a randomized experiment to earn the right to say one word — because. The random assignment was cryptographically fair; we verified it. Weeks in, an audit found that the experiment could not be analyzed at all, and never could have been. The coin was flawless. The join from the coin to the thing it decided did not exist in the data.
To make a randomized experiment analyzable you need three things that have nothing to do with the randomizer: a join that links each draw to the unit it was applied to and that unit's outcome, a guard that makes selective re-drawing impossible or at least detectable, and a power calculation done before you start so you know the sample can answer the question. We had a provably fair coin and none of the three. This is what each one is, why its absence is fatal, and what you actually have to record at the instant of assignment.
Why randomize at all: to earn one causal word
Most of what you can measure about a running system is observational — patterns under whatever policy happens to be in force. Observational data can generate hypotheses, but it cannot, by construction, tell you that a change caused an outcome, because the thing you changed is tangled up with everything else that moved at the same time. There is exactly one clean escape: assign the treatment at random. Randomization is what severs the treatment from all the confounders and lets you say the forbidden word — because — about the difference between arms.
