The Bloom filter that never existed, and the two ceilings it was hiding

This is a submission for DEV's Summer Bug Smash: Smash Stories powered by Sentry.

The most expensive bug I fixed this year was not in the code. It was in the documentation, and it had been shaping what everyone believed the code did.

The setup

HydraDNS is an open-source DNS security gateway I build in Go. Router points at it, it filters every DNS query on the network against a 92k-domain blocklist, blocks the bad ones, forwards the rest. Before putting it on anyone else's network I wanted a real number for what one box could take, so I sat down with dnspyre and a rule I had written for myself: every number becomes a sales claim or a fix ticket. No number, no claim.

Our feature sheet said the blocklist was backed by a Bloom filter, sub-millisecond lookups. Here is the uncomfortable part: at every load this system had ever run, that claim was indistinguishable from the truth. Normal-traffic latency sat at one or two milliseconds. There was nothing to doubt, because nothing observable disagreed.