If you have Cloudflare in front of your site, and at some point you wrote a cache rule that matches bots, go and read that rule again. I wrote one to be helpful to AI crawlers. What it actually did was store my error responses at the edge and replay them to every AI crawler for an hour at a time.
I found out because OpenAI's crawler could not read my site. GPTBot was getting HTTP 429. Amazonbot and Bytespider were getting 403. I run taghazout.io alone, so there was nobody to hand the problem to. I did what most people do first: I looked at the response headers.
That is where I got it wrong.
The header I trusted
The 429 responses carried this:
