IBM's Benchpress pins the seeds that build its circuits and passes no seed to the code that compiles them. On one circuit, over 400 seeds per version, the real change between two Qiskit releases is +5.37%. The suite's own three-run protocol reports it as a ≥10% regression 24.4% of the time. Twenty runs per version — about forty hours of compute — still leaves 3.7%.
A benchmark suite is an instrument. You point it at two versions of a compiler and it tells you whether the new one got worse. Somebody then merges or reverts a pull request on the strength of that answer.
So the question I care about is not "is this compiler slower". It is how often does the instrument give an answer that disagrees with its own long-run behaviour. I call that finite-sample decision risk: the probability that a verdict computed from k runs per version disagrees with the verdict implied by the mean of the same measurement.
Where the randomness comes from
Qiskit's transpiler is stochastic. Its routing pass uses randomness, and generate_preset_pass_manager takes a seed_transpiler argument to pin it.
