llms.txt is a proposed convention: a Markdown-formatted file served at /llms.txt that tells AI systems which pages on your site are worth reading and what each one covers. It is a curation file rather than an access-control file — where robots.txt says what a crawler may not fetch, llms.txt says what is worth fetching.
The motivation is context budget. A model answering a question cannot read an entire site, and much of what it would encounter — navigation, boilerplate, duplicated fragments — is noise. A short, well-organized index of canonical pages, grouped by topic with a sentence of description each, gives a retrieval system a far better starting point than a sitemap of a thousand equally weighted URLs.
What the File Contains
The convention is deliberately simple:
- An
#heading with the site or organization name - A short blockquote summarizing what the site is and who it serves
##sections grouping related pages by theme- Under each, a Markdown link per page with a brief description of what it answers
- An optional
## Optionalsection for material that can be skipped under tight context limits
Some sites also publish llms-full.txt, which inlines the full text of the listed pages rather than linking to them.
llms.txt vs. robots.txt vs. sitemap.xml
| robots.txt | sitemap.xml | llms.txt | |
|---|---|---|---|
| Purpose | Restrict access | Enumerate all URLs | Curate the useful subset |
| Audience | Crawlers | Search engines | AI systems and agents |
| Format | Directives | XML | Markdown |
| Editorial stance | None | None — everything is listed | Explicit; you choose and describe |
| Standardized | Yes, long-established | Yes | No — a proposal, not a standard |
Limitations
- It is not a standard and adoption is not guaranteed. No major AI provider has committed to reading
llms.txt, and there is no verification that any system consumes it. Treat it as low-cost preparation, not a lever with measurable return. - Hand-maintained files rot. The most common failure is a file that lists last year's pages. A stale index is worse than none, because it actively points systems at the wrong content.
- It does not create authority. Curating your pages does not make a model trust them — that is a GEO problem.
- It is not access control. Anything sensitive belongs behind authentication, not omitted from a text file.