19% of the top 2,000 sites block AI crawlers On 2 August 2026 I scanned the top 2,000...
Audit of top-2,000 sites: 19% block AI crawlers; blocking is categorical (75% reject all four), driven by site type not ranking. Yet 136 sites allow in robots.txt but block at edge (WAF/CDN)—policy mismatch risks deals; external verification required.
19% of the top 2,000 sites block AI crawlers
On 2 August 2026 I scanned the top 2,000 domains from the Tranco top-1M list and
asked one question of each: can an AI crawler read this site?
Not "should it" — that's a business decision every site gets to make. Just:
what does the site actually do when GPTBot, ClaudeBot, PerplexityBot, or
Should websites allow AI search crawlers? Not blindly. Blocking every AI crawler can protect...
Sorting the 15 AI crawlers by what blocking each actually costs you — not another copy-paste list. Corrects the two token…
Three weeks ago I wrote about the difference between AI training crawlers and AI...
Recently I came across a discussion about AI crawlers and robots.txt files. The conversation centered...
Major publishers have increasingly restricted AI crawler access to their websites, turning a once...
Everyone's arguing about AI search visibility. Almost nobody is measuring the boring part: whether AI...
