I rewrote the same detector three times in one day. Each fix introduced a new way of being wrong.

I wrote a post last week about an LLM fabricating a company's official website. Then I built a detector to find more cases at scale.

I have now rewritten that detector three times in one day. Each version fixed the previous version's bug and introduced a new way of mislabelling the same data. The counts went 1 → 134 → 41 → 12.

The interesting part isn't the final number. It's that all three bugs were the same bug wearing different clothes, and I had published an article about that exact bug in between versions two and three.

Version 1: blind to agreement

The detector's job: find domains an LLM calls a brand's "official website" when the brand doesn't own that domain.

I rewrote the same detector three times in one day. Each fix introduced a new way of being wrong.

Related reading

Seven things I got wrong about my own project in one day, and what caught each…

I built a detector for invisible records on Monday. On Tuesday it caught me,…

I built a detector that hit 100% accuracy. Then I spent a day trying to prove…

I killed six of my own results in one night. That was the win

301 duplicate IDs in the browser, 0 on the JVM: one real bug, end to end

My bot detector said 95% fake. It agreed with me instantly — that was the tell.