Originally published on the Lyra blog.
An llms.txt file is a plain Markdown file you host at /llms.txt that hands AI crawlers a clean, curated map of your site's most important pages. It is not styled for people. It is a shortlist a language model can read in one pass: your site name, a one-paragraph summary, and a few sections of links with short descriptions. The idea, proposed by Jeremy Howard in 2024, is simple. Models work better with context than with raw HTML, so give them the context directly.
This guide covers what goes in an llms.txt file, whether it actually moves the needle, how it differs from robots.txt and a sitemap, and how to write and host one in a few minutes.
What is llms.txt and what goes in it?
llms.txt is a Markdown file at the root of your domain that points AI models at the pages that matter, with a sentence of context on each. A browser will render it as text. A model reading it gets a quick, structured sense of what your site is and where the good parts live, without crawling and parsing every page first.
