llms.txt is a proposed convention: a single Markdown file at the root of a domain that gives large language models a curated, human-written map of the site's most useful content. It is a hint, not a protocol — nothing enforces it, and support varies.
What it is not
- Not robots.txt. robots.txt controls access. llms.txt suggests what matters to a model that already has access.
- Not sitemap.xml. A sitemap is exhaustive and machine-generated. llms.txt is selective and editorial — the twenty links you would hand a new analyst, not the twelve thousand URLs you have.
- Not a ranking factor. Publishing one does not make an engine cite you.
The file format
The convention is deliberately small. A conforming file is Markdown, served at /llms.txt as text/plain or text/markdown, and structured like this:
- A single
#H1 with the site or project name. - An optional blockquote immediately after it, giving a one-paragraph summary.
- Optional free prose giving context.
##sections, each containing a list of links in the form- [Title](url): short note.- An optional final section named
## Optional, holding links that may be skipped when context is tight.
Some sites also publish /llms-full.txt, which inlines the full text of the linked documents into one file so a model can ingest everything in a single fetch.
Why anyone bothers
Three reasons hold up:
- Context budgets are small. A model reading your site has limited room. Pointing it at your ten best pages beats letting it sample randomly.
- Documentation is hard to crawl well. Nav-heavy, JavaScript-driven doc sites degrade badly in text extraction. A curated list of clean URLs sidesteps that.
- It forces an editorial decision. Writing one makes you state, in one paragraph, what your site is for. That statement is useful far beyond the file.
The adoption reality
Be clear-eyed: as of now no major AI provider has publicly committed to reading llms.txt as part of its retrieval pipeline. Adoption is strongest among developer-tool and documentation sites, and the strongest verified benefit is for agents and assistants that are explicitly pointed at the file. Treat it as low-cost, low-risk, potentially useful — not as a growth lever.
Should you publish one?
Yes, if you have documentation, a product with non-obvious concepts, or a large content library where the important pages are not the most-linked ones. The cost is an hour. The risk is publishing a stale file, which is worse than none — treat it as content, and review it when your information architecture changes.
Skip it if your site is five marketing pages. A model will read all five anyway.
Common mistakes
- Dumping the whole sitemap into it. The point is curation.
- Using relative URLs. Use absolute ones; the file may be read out of context.
- Serving it as HTML, or returning a 404 with a 200 status.
- Writing link titles that repeat the URL instead of describing the page.
- Never updating it.
The next guide walks through writing and validating one line by line.