We finally opened Cloudflare's AI Crawl Control last week, mostly out of curiosity about which AI bots actually visit us. It had been collecting quietly the whole time. The first 24-hour window we looked at showed 1,890 requests from AI agents, and 483 of them failed. Compared with the previous day, that failure count was up 63 percent.
The worst record belonged to the agent we were most curious about. ChatGPT-User is the one OpenAI documents as user-initiated: it visits a page when someone asks ChatGPT or a Custom GPT a question, rather than crawling on a schedule. It accounted for 859 requests. Only 412 came back 200.
So roughly half the requests wearing that name were bouncing. I wrote it up as a redirect problem: we had migrated docs slugs more than once, so stale URLs would explain it, and a redirect map would fix it. Clean story. It was wrong.
Then I looked at the actual paths.
/id_rsa
