We ran a small audit last winter because a client asked whether GPTBot could "see" their new help centre. Search Console looked fine. PageSpeed Insights on the homepage looked fine. Server logs told a different story: long tail URLs timing out, a category template returning a JavaScript shell on the first response, and a /robots.txt rule that blocked one path the marketing team had already pitched for AI Overviews.
Nothing in that list was exotic. It was the kind of drift you only notice when you stop testing the three URLs everyone bookmarks and start reading what bots actually request.
That audit changed our monitoring priorities more than any slide about "optimising for ChatGPT." Bots read fast pages too. They also abandon slow ones, skip empty HTML, and respect robots.txt literally. Below is what we reprioritised, what we stopped overclaiming, and where scheduled PageSpeed monitoring fits once crawlability is on the board.
Why we audited GPTBot and other AI crawler traffic in server logs
Large language models and AI search products reach your content through partner indexes and dedicated crawlers. OpenAI publishes GPTBot; Google still sends Googlebot for Search and related features. Other vendors document their own user-agents. The exact mix varies by site and industry.
