I have been running a small measurement rig for two weeks: a set of scripts that load our pages in a fresh anonymous browser context and record what is actually there. Directory listings, articles, search positions. It has been genuinely useful and it has been wrong in one specific way, six separate times, and the six are the same mistake.
Every time, a script failed to read something, and I wrote down the failure as a fact about the world. The last two happened while I was drafting this post, which is the part I would keep if I could only keep one.
One: I turned four observations into a property
Two directories were serving an anti-bot challenge instead of their content. I checked, got the challenge, checked again the next day, got the challenge, and after four identical observations I recorded it in my notes as an established property of those sites: they block automated readers.
On the fifth check, both loaded perfectly. Nothing had changed on my side.
