Last week I published a post about nine measurement bugs I found in my own evaluation harness, all of which happened to favour my results. Two readers replied with things I hadn't found.
Both were about my checks rather than my numbers. I verified both and fixed both. My results did not change at all.
That turned out to be the interesting part.
Reader one: a canary that could not detect what it claimed to
Some setup. The harness mutates a source file in a temp copy and runs the test suite against it. To prove that mutations actually reach the interpreter, I had a canary: overwrite the module with unparseable garbage and assert the suite fails.
