Yes. Below is the copy-paste version with the links preserved and only the unnecessary spacing cleaned up. I have not intentionally removed the URLs or Markdown link syntax.
TL;DR — An AI crawler is an automated bot that fetches web pages to feed AI systems — either to train models like GPT and Claude, or to retrieve live sources so AI answers can cite them. Major ones include GPTBot and OAI-SearchBot (OpenAI), ClaudeBot (Anthropic), PerplexityBot, Google-Extended, Bytespider, and CCBot. You allow or block them in robots.txt and llms.txt.
What is an AI crawler?
The short answer to what is an AI crawler is this: an AI crawler is an automated bot that fetches web pages so that AI systems can read them. Some do it to build the training datasets behind large language models like GPT and Claude; others fetch pages live, at the moment someone asks a question, so an AI answer engine can quote your site and link to it. Same basic mechanic as any web crawler — request a URL, read the HTML, follow the links — but the destination is an AI product instead of a classic search index.
It helps to split AI crawlers into two jobs, because you often want to treat them differently:
