Definition
llms.txt is a Markdown file published at the root of a website — example.com/llms.txt — that gives an AI system a curated, human-written map of what the site contains and where the useful pages are. Jeremy Howard proposed it on 3 September 2024.
The single most useful fact about it is one that most articles bury: Google does not use it. Google's AI optimisation guide, updated 15 June 2026, says plainly that you do not need to create new machine-readable files, AI text files, markup or Markdown to appear in Google Search including its generative features, because Search itself does not read them. Google has separately confirmed the file will neither help nor hurt rankings.
That makes llms.txt an unusual thing to explain, because the reason people search for it — "will this get me into AI search results?" — has a one-word answer, and it is no. The reason it nonetheless exists, is maintained by serious engineering organisations, and is worth understanding, is entirely different: it is a documentation convention for tools that read your docs on purpose, rather than a signal for crawlers that find them by accident.
How It Works
The problem the spec names in its own opening is a size problem: context windows are too small to hold most websites. That is worth working through with numbers, because the ratio is the whole argument.
Take a typical documentation page. The rendered HTML carries navigation, a sidebar, a cookie banner, analytics scripts, inline CSS, and a footer, and the actual prose might be a fifth of the bytes. A 200 KB HTML page is roughly 50,000 tokens once tokenised. The same content as clean Markdown might be 8 KB, or about 2,000 tokens — call it a 25× reduction, though the real figure varies enormously by site.
Now scale that to a task. An agent answering a question about your API might need to consult twenty pages. At 50,000 tokens each that is 1,000,000 tokens, which exceeds or saturates most context windows and costs real money to process. At 2,000 tokens each it is 40,000 tokens — a rounding error in a modern context window, and roughly 4% of the token spend.
That is the mechanism. llms.txt does not make a model smarter about your site; it makes your site cheap enough to read in full.
The format is deliberately minimal. The spec requires exactly one thing — an H1 naming the project — and everything after it is optional:
# Acme API
> Payments infrastructure for developers. REST and webhooks, with SDKs
> for six languages.
Notes for anyone reading this: the v1 API is frozen. Use v2 for new work.
## Docs
- [Quickstart](https://acme.dev/docs/quickstart.md): auth and first request
- [Webhooks](https://acme.dev/docs/webhooks.md): event types and retries
## Optional
- [Changelog](https://acme.dev/changelog.md)
After the required H1 come, in this order: an optional blockquote summary; zero or more Markdown sections of any type except headings; and zero or more H2-delimited sections containing file lists. A file list entry is a Markdown link, optionally followed by : and a note.
Two things follow from that structure that people get wrong.
It is a content file, not a directive file. The comparison everyone reaches for is robots.txt, and it is misleading in both directions. robots.txt is access control — it tells a crawler what it may fetch, and every major crawler honours it. llms.txt tells nobody to do anything. It is an offer, and consuming it is entirely optional. There is no enforcement, no user-agent matching, no Disallow.
The links should point at Markdown, not HTML. An llms.txt whose entries all resolve to normal HTML pages recreates the size problem it exists to solve. The convention that makes it work is serving a .md version of each page alongside the rendered one.
llms-full.txt is worth a note because it is widely referenced and is not in the specification. The spec describes only /llms.txt. The FastHTML project shipped a variant called llms-ctx-full.txt that inlines the content of the linked files rather than linking to them, and an llms-full.txt convention grew from that. Tools support it inconsistently, because no standard defines it.
Real-World Applications
The gap between who publishes llms.txt and who reads it is the most instructive thing about the file, and both halves are measured.
Publishing is real but concentrated. As of June 2026, about 8.7% of the world's top 1,000 websites publish one. A separate 300,000-domain sample put adoption at 10.13% — roughly 30,000 domains. The sites doing it are overwhelmingly developer-facing: Stripe, Cloudflare and Anthropic all ship one, and the pattern across the adopting cohort is "company whose product is an API".
Reading, measured by web requests, is close to zero. Ahrefs tracked 137,210 domains with an llms.txt file and found that 97% received no requests for it at all during May 2026. That leaves roughly 4,100 domains — three in a hundred — whose file was fetched even once in a month.
Those two findings are usually presented as a contradiction, or as proof the whole idea failed. They are neither, because they measure different things. The 97% figure counts HTTP requests to the file from crawlers. It does not count the ways llms.txt is actually consumed:
- Coding agents pull it as documentation. Cursor and GitHub Copilot can be pointed at a library's docs, and an llms.txt is the cheapest possible entry point for that — one fetch, initiated by a developer working on that library, not by a crawler sweeping the web.
- Assistants retrieve it during a session. Perplexity and Claude will fetch it when a user's question is about that specific site. Again: demand-driven, one request, invisible in a crawl-frequency study.
- Doc platforms generate it automatically. Mintlify and similar tooling emit
llms.txtas a build artifact, which is why adoption among API companies is high without anyone treating it as a marketing decision.
So the honest summary is that llms.txt is a pull mechanism for a narrow, high-intent audience — someone or something already trying to read your documentation — mistakenly marketed as a push mechanism for broad AI visibility. Judged as the first, it works. Judged as the second, the 97% figure is exactly what you would expect.
Challenges
The premise most people adopt it on is false. The dominant search intent behind this term is "will llms.txt get me cited in AI answers", and for Google the documented answer is no. Publishing one as an SEO tactic is not harmful, but it is a null action, and the hours are better spent on content an answer engine can actually quote — which is what Generative Engine Optimization is about.
It is a second copy of your site that can drift. The file lists URLs and describes them. Rename a page, retire an endpoint, restructure your docs, and the llms.txt keeps confidently pointing at what used to be there. Unlike a broken link on a rendered page, nobody sees this — the file has no UI and, per the adoption data, usually no visitors either. A stale llms.txt is worse than none, because it feeds an agent wrong information with the authority of an official file.
Nothing validates it. There is no schema, no linter in most stacks, no Search Console report. A malformed file fails silently.
It leaks your structure to anyone who asks. The file is a deliberate, curated index of what is worth reading on your site. That is the point, and it is also a decision: you are publishing a machine-readable map for scrapers as well as for agents you like. robots.txt at least lets you distinguish user agents; llms.txt does not.
The llms-full.txt ambiguity is a real interoperability problem. Because it is a convention rather than a spec, two tools can disagree about whether it should exist, what it should contain, and whether it supersedes llms.txt.
Future Trends
The interesting question is not whether adoption grows — it will, because doc platforms generate the file automatically and that alone drives the number up without any deliberate decisions. The question is whether the consumption side ever standardises.
Right now llms.txt occupies an awkward position next to the Model Context Protocol. MCP solves a strictly larger version of the same problem — giving a model structured, authoritative access to a system — with an actual protocol, a handshake, and tool definitions. A documentation site can expose an MCP server instead of a text file, and get queryable access rather than a static list. Where MCP is available, it is the better answer, and the pressure on llms.txt is that it may end up as the fallback for sites that cannot justify running a server.
The counter-argument is cost. An llms.txt is a build artifact that costs nothing to serve and nothing to operate. That is a durable advantage over anything requiring a running process, and it is why the file is likely to persist in the developer-documentation niche even if it never becomes a web-wide convention.
What is unlikely to change is Google's position. The stated reasoning — that Search does not need a separate machine-readable surface because it already renders and understands the page — applies to any future file of this shape, not just this one.
Note: this term was last reviewed in August 2026. The adoption percentages, the Ahrefs request-rate figure and Google's stated position are dated to that review and are the parts of this page most likely to age.