We nearly published a 2x regression that didn't exist

We were about to ship a performance write-up claiming three queries had gotten 2-4x slower. All three numbers were wrong, and the reason is one every benchmark author should worry about: the same binary, on the same machine, running the same query, gave different answers depending on what time of day we ran it.

The measurement that didn't add up

While checking a change to Samyama Graph, we timed LDBC SNB Interactive query IC9 on one commit, one dataset, one 16-core workstation, four hours apart:

when

commit

We nearly published a 2x regression that didn't exist

Related reading

Your offline LLM eval isn't measuring your model — it's measuring your rate…

Three dumb ways our prod got slow (and not one was a slow algorithm)

I A/B tested 4 LLMs on the same 500 queries. The results surprised me.

Six queries, three runs, every mean 8 — and the fine-tune wasn't why

I built a page that only publishes numbers surviving our own checks. It caught…

SurrealDB 3.x by the numbers