Most lists that claim to help you block AI crawlers are copy-pasted and dangerously wrong about the two tokens that actually matter. Sorting them properly is not an abstract taxonomy exercise. It is the single decision that determines whether your content vanishes from AI answers while training continues — or vice versa.
We maintain the crawler registry in lib/ai-crawlers.ts that powers techpotions’ free AI robots.txt generator. Every agent string and description was verified against the operator’s own crawler documentation. The registry holds 15 verified bots across four categories, and that four-way split is this article’s structure, because the categories map directly to what blocking costs you.
Two tokens almost everyone gets wrong
Before the list, the single most important correction to make, and almost every listicle on this query gets it wrong: Google-Extended and Applebot-Extended are not crawlers. They are robots.txt tokens — product controls that govern whether your content is used for Gemini and Apple foundation-model training.
Blocking Google-Extended does not affect Google Search crawling, Google ranking, or regular Applebot search indexing. People block them believing they are opting out of AI Overviews, and are actually opting out of nothing they think they are, while leaving search indexing completely untouched.
