Originally published on the Merlonix blog.
/llms.txt is the file everyone adding "AI SEO" to their site keeps hearing about and nobody quite explains. It gets lumped in with robots.txt because it lives in the same place and has a similar name, so people assume it's another access-control file — a way to let the good AI crawlers in or keep them out. It is not. robots.txt says who may fetch what. llms.txt says here is what matters, and where it is. One is a gate; the other is a table of contents. Adding an llms.txt grants no access and blocks nothing — a crawler that a WAF or robots.txt rule is keeping out will never see it, and a crawler that's already allowed in doesn't need your permission to read it. What it does is make a site legible to an agent that has to otherwise reverse-engineer your navigation to find the three pages that actually answer the question.
Here's the whole thing: what the file is, the exact format, what to put in it, the honest answer to "does anything actually read it yet," and how to confirm yours is valid rather than just present.
What llms.txt Actually Is
llms.txt is a proposed convention — published at llmstxt.org in late 2024 — for a single Markdown file at your domain root (https://example.com/llms.txt) that curates your most important content for a large language model. The premise is simple: an LLM working with a limited context window can't crawl and hold your entire site, and rendered marketing HTML is noisy — nav, cookie banners, footers, scripts. A short, hand-curated Markdown file that names your key pages and links straight to them is far cheaper for an agent to consume than making it guess which of your 400 URLs are the canonical docs.
