28 AI crawlers with operator, purpose, and observed robots.txt compliance — JSON + CSV, CC BY 4.0, no signup. Here's why the "purpose" field matters more than the list itself.
While building AI-crawler tooling I kept needing the same thing: a current,
machine-readable list of AI crawler user agents. Every blog post has a partial
list, half of them outdated, none of them structured. So I published mine as an
open dataset.
GitHub: https://github.com/osamamumtaz01/ai-crawler-user-agents
Everyone's arguing about AI search visibility. Almost nobody is measuring the boring part: whether AI...
Sorting the 15 AI crawlers by what blocking each actually costs you — not another copy-paste list. Corrects the two token…
I wanted to know which sources ChatGPT, Claude, Gemini and Perplexity actually read to answer the same question. Extracting that…
Recently I came across a discussion about AI crawlers and robots.txt files. The conversation centered...
19% of the top 2,000 sites block AI crawlers On 2 August 2026 I scanned the top 2,000...
More and more answers reach your readers through ChatGPT, Claude and Perplexity instead of a blue...
