llms.txt

llms.txt is a plain markdown file that acts as a reading list for language models. Where a sitemap lists every URL for a search engine, llms.txt picks the handful of pages that explain what a product or project is, with a line on each, in a format a model can take in within a small context window.

It is a community proposal, not a standard with an enforcement body, and it controls nothing. It does not allow or block crawling; that remains the job of robots.txt. Its value is editorial: you decide which pages describe you, rather than leaving a model to infer it from whatever it happened to fetch.

When an assistant describes your company, it works from what it can read quickly. A clear summary and a curated list of your key pages is cheap to publish and removes guesswork about pricing, positioning and what the product does.

Adoption is uneven even among the companies selling AI visibility. In our scan, about half of the vendors publish one, which makes it a small, visible way to look prepared. Treat it as good documentation hygiene rather than a ranking lever: whether a given engine reads it is that engine's decision.

• The llms.txt proposal is by Jeremy Howard, "Published September 3, 2024", and specifies "An H1 with the name of the project or site. This is the only required section", followed by "A blockquote with a short summary of the project" and "Zero or more markdown sections delimited by H2 headers, containing 'file lists'".Jeremy Howard, The /llms.txt file (llmstxt.org), 15 Sept 2026
• The proposal says "The 'Optional' section is used, by convention, for secondary information: links an agent can skip when a shorter context is needed."Jeremy Howard, The /llms.txt file (llmstxt.org), 15 Sept 2026
• Of 15 AI search and content vendors scanned, our own included, 8 publish an llms.txt, 6 do not, and 1 could not be checked because the server refused the request.Veritas crawler policy scan of 15 AI search vendors, 15 Sept 2026

1. 1

   Start with an H1 naming the site or product, then a one-paragraph blockquote saying what it is, who it is for and what it costs, in plain sentences.

2. 2

   Add H2 sections of links, each written as a markdown link with a short note: product pages, pricing, documentation, and the pages that answer the questions buyers ask most.

3. 3

   Put nice-to-have links under an H2 titled Optional, so an agent short on context knows what it can skip.

4. 4

   Publish it at /llms.txt as plain text, link to real canonical URLs, and keep it in step with pricing and product changes, because a stale summary is worse than none.

5. 5

   Validate the structure before publishing, and recheck it whenever key pages move.

• Dumping every URL from the sitemap into it, which defeats the point of a short, curated list.
• Expecting it to block or allow AI crawlers. Access is controlled by robots.txt, which llms.txt does not change.
• Assuming every AI engine reads it. It is a proposal, so check each vendor's own documentation rather than relying on it for visibility.
• Letting the summary drift out of date, so a model repeats an old price or a retired product.

robots.txt

robots.txt tells crawlers what they may fetch and is widely honoured. llms.txt suggests what a model should read first and grants or blocks nothing.

XML sitemap

A sitemap lists every indexable URL for search engines. llms.txt lists a curated few with human-readable notes, written for a model with limited context.

1. 1Jeremy Howard, The /llms.txt file (llmstxt.org) read 15 Sept 2026
2. 2Veritas crawler policy scan of 15 AI search vendors read 15 Sept 2026