More and more answers reach your readers through ChatGPT, Claude and Perplexity instead of a blue...
Major publishers (Reuters, nytimes) accidentally block retrieval agents (ChatGPT-User, Claude-SearchBot) in robots.txt, losing AI citations. Silent disappearance from LLM answers = zero traffic from growing AI-search; competitive loss disguised by misconfiguration.
More and more answers reach your readers through ChatGPT, Claude and Perplexity instead of a blue link. Whether those answers can ever cite you is decided by one file: robots.txt. And most robots.txt files decide it by accident.
Two kinds of AI crawlers — and only one of them matters for citations
Kind
User agents
Blocking it means
Most AEO/GEO advice hands you a checklist: add structured data, write answer-first, put a date on it,...
Your robots.txt lists User-agent: GPTBot - Allow: /. The page loads fine in a browser. The "AI...
Recently I came across a discussion about AI crawlers and robots.txt files. The conversation centered...
Ask ChatGPT, Perplexity, or Google's AI Overviews a real question and you get one synthesized answer...
When a person asks ChatGPT, Perplexity, Claude, or Gemini a question, the assistant answers from...
Sorting the 15 AI crawlers by what blocking each actually costs you — not another copy-paste list. Corrects the two token…
