A lesson we’ve learned in experimentation at Datadog is how easy it is to fall into interpretative pitfalls even when following rigorous conventions. For example, consider an experimentation program that appears to do everything right on the surface. The team pre-determines experiment sample sizes to provide a small minimal detectable effect (MDE) for a product change, waits until those sample sizes have been reached to make decisions, and ships only experiments that are statistically significant at the 95% confidence level.
The problem becomes apparent when the team looks at multiple experiments in aggregate. Recall what a 95% confidence level guarantees: Even when a change has no true effect, the test will still flag it as significant 5% of the time purely by chance. If only 5% of the experiments are statistically significant, the program has the same rate as one that tests exclusively changes that have zero true effects. In such a scenario, every winning experiment is a mirage despite the team using statistical methods that control the false positive rate (under the null hypothesis).
This example illustrates the importance of the effect distribution, or the distribution of true effects across a set of related experiments. In our previous example, the results are consistent with an effect distribution that is a point mass concentrated exactly at zero, which is equivalent to the distribution under the null hypothesis of no effect. In this case, it’s clear that the experimentation program would be chasing its own tail because there are no true wins, and common statistical methods fail us because any significant result is a false positive. An individual significant result might seem convincing in isolation, but it’s far less so with the context that 5% of the tests from the program are significant.
