Your firewall is your AI policy — I probed 18 major sites to read it

Everyone's arguing about AI search visibility. Almost nobody is measuring the boring part: whether AI crawlers can fetch your pages at all.

So I built a small open-source tool — geo-crawl-audit — that probes any site with the user-agents of every AI crawler that matters (GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot, ChatGPT-User, and friends), measures how each is treated versus a normal browser, and checks the thing most people never think about: how many words exist in the raw HTML before any JavaScript runs. Because here's the detail the industry keeps missing — GPTBot, ClaudeBot, and PerplexityBot don't execute JavaScript. For most sites, Googlebot (feeding Gemini) and Applebot are the only AI-adjacent crawlers that render it. A site can rank #1 in Google and be a blank page to nearly everything else.

I pointed it at 18 major sites on August 7. Five findings worth your time.

1. Access patterns line up with the business relationships

The Guardian — which has a content deal with OpenAI — serves my simulated GPTBot, OAI-SearchBot, and ChatGPT-User a clean 200. The same request wearing ClaudeBot, PerplexityBot, or CCBot: 403, and those names are in its robots.txt disallow list too. Policy and enforcement agree.