A client called me last month, half panicked, after getting an email from his hosting provider. The subject line wasn’t a good one: “Your site is consuming excessive resources." The options were either to upgrade the plan or migrate to a better server.
His traffic graphs in GA4 were flat. Same visitors as the previous month. So I went into the access logs to understand what was happening.
It wasn't a DDoS. Well, it was kinda a DDoS. Made by AI bots.
The crawler invasion
Pull up the raw access log of any moderately popular site right now, and you'll see them: GPTBot. ClaudeBot. PerplexityBot. Bytespider. Google-Extended. Meta-ExternalAgent. OAI-SearchBot. CCBot. Applebot-Extended. Each one of them crawling page after page, product after product, post after post, on a schedule none of us were consulted about.
