You put a robots.txt on your site to tell search crawlers what to ignore. You add a sitemap.xml to help them find everything. These standards work because crawlers visit your site repeatedly — on a schedule, automatically, indefinitely. Instructions you leave in files become part of an ongoing conversation between your server and the crawler.
llms.txt doesn't work like that. That's the thing most articles about it miss, and it's the reason the standard is simultaneously more limited and more interesting than it sounds.
What llms.txt Is
llms.txt is a proposed standard — not officially adopted by any major AI provider — created by Jeremy Howard from Answer.AI. The idea: place a Markdown file at your domain root (yourdomain.com/llms.txt) that describes your site and lists your important pages with brief descriptions. Clean, human-readable, structured for AI consumption rather than HTML parsing.
A minimal example:
