AI crawler user agents are self-reported: 468 real fetches, 991 fake ones

I used to watch AI crawler traffic on my site as a table grouped by User-Agent: so many requests from ChatGPT-User, so many from GPTBot. Numbers going up meant the AI systems were picking the site up.

Then I re-cut eight days of logs by verification result. Requests that actually fetched an article: 468. Requests probing for .env and friends: 991. Of everything calling itself GPTBot, Cloudflare could verify 13% as OpenAI.

Here is how to separate the impersonators on a Cloudflare free plan, and what the numbers looked like.

A User-Agent is a claim, not evidence

Writing GPTBot/1.2 into a header costs nothing, so a table grouped by UA is a list of claims.

AI crawler user agents are self-reported: 468 real fetches, 991 fake ones

Related reading

I split my site's AI bot traffic three ways. Only one turns into a citation.

19% of the top 2,000 sites block AI crawlers — and 136 more block them without…

Your firewall is your AI policy — I probed 18 major sites to read it

How to Verify an AI Crawler Is Who It Says It Is

I let every AI crawler in. A month later I cannot tell you how many people read…

AI Bots Are Reading Your Site. Here's How to Make Them Sell You.