What llms.txt is for
AI assistants and coding tools read websites differently from people. They have a limited context window, and pages full of menus, scripts and cookie banners waste it. The llms.txt proposal, published by Jeremy Howard at llmstxt.org in September 2024, offers a fix. Put one Markdown file at /llms.txt that explains in a few lines what your site is and links to the pages that matter, preferably as clean Markdown.
It is a community proposal, not a formal standard. It doesn’t replace your sitemap, which lists every page for search engines, or robots.txt, which controls what crawlers may fetch. It is a short, hand-picked reading list for AI.
The format
The file is Markdown with a fixed order. Only the first part is required:
- An H1 title, with the name of the site or project. Required.
- A blockquote summary, one or two sentences with the key facts.
- Optional details: paragraphs or lists with anything an AI should know first (terminology, versions, limits). No headings here.
- H2 sections, each containing a list of links in this form:
A section named Optional has a special meaning: its links can be skipped when an AI needs a shorter context. Put the changelog, archives and secondary pages there.
The proposal also suggests a Markdown copy of each page at the same URL with .md added: /docs/intro.html becomes /docs/intro.html.md, and a folder URL like /docs/ becomes /docs/index.html.md. Tick Link to Markdown versions when importing a sitemap to write the links that way. Only do that if those files exist on your site.
Worked example
Click Load example to see a complete file for a fictional weather API:
# Acme Weather API
> Acme Weather API gives current conditions, hourly forecasts and historical data for any location, over a simple REST API with JSON responses.
All endpoints need an API key sent in the `X-Api-Key` header.
## Docs
- [Quick start](https://example.com/docs/quick-start.md): Get a key and make your first request in five minutes
- [Forecast endpoint](https://example.com/docs/forecast.md): Hourly and daily forecasts, parameters and response fields
## Optional
- [Changelog](https://example.com/changelog.md)With this file, an assistant asked “how do I get a 7-day forecast from Acme?” can read the summary, pick the forecast page from its note and load just that page, instead of crawling the whole site.
Prefilling from your sitemap
Most sites already have a sitemap.xml. Open it in your browser, copy the XML and paste it into Prefill links from a sitemap.xml. The tool reads every <loc> URL, skips the home page, and groups the rest by their first folder. /docs/getting-started goes under Docs, /blog/launch under Blog, and top-level pages under Pages. Titles come from the URL (“getting-started” becomes “Getting started”). Nothing is fetched: you paste the XML yourself, so this also works for staging sites.
If you paste a sitemap index (a list of other sitemaps), the tool tells you to open one of the child sitemaps instead.
Then curate. A good llms.txt usually has 10 to 50 links, not 2,000. Delete pages an AI doesn’t need, such as tag archives, legal boilerplate and old news. Write a short note for each link that remains, because the notes are what an assistant uses to choose.
The validator
The live check under the output, and the Validate tab for existing files, test the rules of the format:
| Check | Severity |
|---|---|
| Missing H1, more than one H1, or text before the H1 | Error |
| H3 or deeper headings, prose inside a link section | Warning |
List items that aren’t [name](url) links |
Warning |
| Relative URLs, which an AI may read out of context | Info / warning |
| Duplicate URLs, empty sections, no summary | Info |
Edit in the builder loads a pasted file into the form, so you can improve an existing llms.txt instead of starting from scratch.
Tips
- Write the summary for a newcomer. Say what the product does, for whom, and any naming that could confuse, such as “Pro” meaning a plan rather than a model.
- Link to the cleanest version of each page. If you don’t serve Markdown yet, convert key pages with document to Markdown.
- Keep it small enough to load. Check the file and the pages it links to with the LLM token counter.
- Pair it with robots.txt. llms.txt invites AI tools in; it doesn’t stop anyone. To block AI training crawlers while keeping AI search, use the robots.txt generator.