llms.txt explained: the format, llms-full.txt and who reads it

Jeremy Howard of Answer.AI published the llms.txt proposal at llmstxt.org in September 2024. The problem it targets is simple: websites are built for people, and a typical page is mostly navigation, scripts, cookie banners and layout. A language model trying to answer a question about your product has limited context and gains little from all that.

The proposal asks sites to publish a plain Markdown file at /llms.txt that says, in a few lines, what the site is and where the useful material lives. It also suggests offering a Markdown version of important pages at the same URL with .md appended, so a tool can fetch clean text instead of HTML.

The intended use is at inference time rather than training. When someone asks an assistant about your API, or points a coding tool at your docs, the tool can read llms.txt first and go straight to the right pages.

People search for it as "llms txt" as often as "llms.txt", and it gets described as "robots.txt for AI". That comparison is misleading. robots.txt is about permission; llms.txt is closer to a curated table of contents. It doesn't control anything.

The format is ordinary Markdown in a fixed order:

1. An H1 with the name of the project or site. This is the only required part.
2. A blockquote with a short summary holding the key facts someone needs to understand the rest.
3. Optional paragraphs or lists with more detail (no headings).
4. H2 sections, each a list of links written as [name](url): optional notes.
5. An optional H2 called Optional, for links a tool can skip when it is short on context.

A small example:

# Acme Invoicing

> Acme is invoicing software for UK freelancers. It creates VAT-compliant invoices, chases late payments and syncs with Xero.

Acme runs in the browser and on iOS. All prices are in GBP.

## Docs

- [Getting started](https://acme.example/docs/start.md): create an account and send a first invoice
- [VAT rules](https://acme.example/docs/vat.md): how Acme handles VAT and the flat rate scheme
- [API reference](https://acme.example/docs/api.md): endpoints, authentication and rate limits

## Product

- [Pricing](https://acme.example/pricing.md): plans and what each includes
- [Changelog](https://acme.example/changelog.md): releases by date

## Optional

- [Company history](https://acme.example/about.md)

Keep the summary factual and specific. It is the part most likely to be read in full, so put the facts a model would otherwise get wrong there: what the product does, who it is for, where it works and how it is priced.

For the link lists, a few habits help:

• Link to canonical URLs, and prefer the .md versions if you serve them.
• Write notes that say what a page answers, not marketing copy.
• Group links the way a newcomer would look for them: docs, product, policies.
• Put anything a tool could safely skip under Optional.

There is no required length, but a file that lists every URL on your site defeats the point. That is what your sitemap is for.

llms.txt is an index: it points elsewhere. llms-full.txt is the opposite idea, the full text of your documentation concatenated into one Markdown file, so a tool can load everything in a single request.

It isn't part of the core llmstxt.org proposal. It grew up as a convention among documentation platforms, and many hosted docs tools now generate both files automatically. Developers use it by pasting the URL into a coding assistant or adding it as a docs source in their editor, which is where it is most clearly useful.

A few practical points:

• Size. A large docs site can produce an llms-full.txt of several megabytes, more than many tools will load. Split it by product area if it gets large.
• Freshness. Generate it at build time from the same source as your docs, never by hand, or it will drift.
• Duplication. It is plain text at a separate URL, so it won't compete with your HTML pages in Google in any way that matters, but you can add an X-Robots-Tag: noindex header if you'd rather it stayed out of search results.

Markdown page copies (/docs/start.md beside /docs/start) do the same job one page at a time. If you already build your site from Markdown, serving these is often a small change.

The two files answer different questions.

• robots.txt says who may crawl what. It is a formal standard (RFC 9309) that well-behaved crawlers check before fetching anything.
• llms.txt says what is worth reading. It grants and denies nothing.

So an llms.txt does not override robots.txt. If you block a crawler in robots.txt, listing pages in llms.txt won't let it in, and if you allow a crawler, leaving a page out of llms.txt won't keep it out.

To control AI access, use robots.txt user-agent groups for each crawler:

User-agent: OAI-SearchBot
Allow: /

User-agent: GPTBot
Disallow: /

User-agent: Google-Extended
Disallow: /

Note that Google-Extended is a control token, not a separate crawler. It tells Google whether content crawled by Googlebot may be used for Gemini training and grounding in some Google products. It does not affect Google Search or AI Overviews.

Get robots.txt right first. A carefully written llms.txt is wasted if your CDN or firewall is quietly blocking the AI crawlers you want.

This is where honesty matters, because a lot of advice oversells it.

Google Search does not use it. Google's Gary Illyes said in 2025 that Google does not support llms.txt and has no plans to, and John Mueller has compared it to the keywords meta tag. Google's guidance for AI Overviews and AI Mode lists no special files.

No major answer engine has confirmed using it for ranking or citations. OpenAI, Anthropic, Perplexity and Microsoft have not said their search products use llms.txt to choose sources. You may see AI crawlers request the file in your server logs, but fetching a file is not the same as acting on it.

Where it does help:

• Coding assistants and agents that a developer points at your docs.
• Tools that let users add a documentation URL as context.
• Anyone pasting your site into a chat and wanting a clean starting point.

That makes llms.txt worthwhile for products with APIs, SDKs or substantial documentation, and low priority for a small marketing site. It costs an hour to write and little to maintain, so the downside is small. Just don't expect it to move your visibility in ChatGPT, Perplexity or Google on its own. Citable content, crawl access and brand mentions do far more; see how to rank in ChatGPT.

A workable process:

1. Write it by hand first. Name the product in the H1, then a two or three sentence summary with real facts: what it does, who it's for, where it works, how it's priced.
2. Link your ten to thirty most useful pages in grouped H2 sections, each with a short note.
3. Serve it at the root as /llms.txt, as plain text (text/plain or text/markdown), with a 200 status.
4. Check it with curl -I https://yourdomain.com/llms.txt and make sure no redirect or bot rule gets in the way.
5. Regenerate or review it whenever key pages move, so it never links to a 404.

If you have documentation, generate llms-full.txt and the .md page copies from your build rather than maintaining them manually.

OpenHelm's GEO audit checks for llms.txt as one item among several: JSON-LD and schema, AI crawler access in robots.txt, citable facts, heading hierarchy and authorship signals. The agent then fixes what it can in the product's repository. More on that at AI visibility.