For twenty years, robots.txt existed to keep bandwidth-hungry crawlers off your server. In 2026, for anyone who wants their work to actually be found, that instinct is backwards.
If your goal is showing up when someone asks Claude, ChatGPT, or Perplexity a question — not just ranking on page one of Google — you need AI crawlers hitting your pages. That's AEO: Answer Engine Optimization, and it runs on a completely different mechanism than classic SEO.
Here's what that actually means in practice, without the hand-waving.
The bots aren't one bot
The biggest mistake I see (and made myself, initially) is treating "AI crawlers" as a single monolith you can allow or block with one User-agent: * rule. They're not. Each provider runs multiple, functionally distinct bots, and blocking one does nothing to the others:
