llms.txt: what it is, whether it works, and how we ship it

llms.txt is a plain markdown file some sites place at their root domain to hand AI systems a short, structured summary: what the site is, and links to the pages worth reading. The honest verdict, as of June 2026: no major AI system has confirmed it actually reads the file or uses it to decide who gets cited. Set it up once as cheap insurance, then stop thinking about it as a ranking lever.

What llms.txt actually is

It is a markdown file proposed by Jeremy Howard of Answer.AI on 3 September 2024, structured as an H1 with the site name, a blockquote summary underneath, and then H2 sections that link out to the pages an AI system should read first. It sits at yoursite.com/llms.txt, written for a model rather than a browser, and it was pitched as a fix for a real problem: full HTML pages, with navigation and ads and scripts, waste a language model’s limited context window.

The original proposal was one file. A companion convention, /llms-full.txt, containing the entire site’s text concatenated in one place, was added later by other implementers, not by Howard’s base spec. Some sites ship both, some ship neither, and the two get conflated constantly in write-ups that treat them as a single standard.

Does llms.txt affect AI search visibility? The honest evidence

No measurable effect has been shown, across two separate studies with different sample frames and different dates. Ahrefs sampled 137,210 domains and published its findings on 15 June 2026: 28% had a valid llms.txt, and of those, 97% received zero requests in May 2026.

SE Ranking data reported by Search Engine Journal on 20 November 2025 covered a different sample of roughly 300,000 domains. There, 10.13% had the file, and there was no statistically significant correlation between having it and how often a domain got cited by AI engines. Only 1 of the top 50 most-cited domains in that dataset had one.

These are not two data points on the same curve. Different sample, different date, different methodology. What they agree on is the direction: having the file is common enough to no longer be a novelty, and having it does not appear to move citation odds either way.

Google’s John Mueller made the comparison explicit in April 2025, likening llms.txt to the old keywords meta tag, a field the industry filled in for years after Google stopped using it. He returned to the topic in June 2026 and said, as reported by Search Engine Journal, that none of the AI systems use it and that it remains speculative for now.

Of the roughly 3% of files that did get requested, the largest AI category was agents and agentic infrastructure at 10.5%, led by Claude-Code. Training crawlers were separate at 5.3%: GPTBot 4.5%, ClaudeBot 0.8%. Retrieval bots, the ones that fetch a page to answer a live question, were 1.1%. All four AI categories together reach 19.5%. The single biggest requester was not AI at all but SEO audit tools at 21.7%, and Ahrefs notes its own crawlers are 10.6 of those points.

What llms.txt does and does not do

Claim made about the file	What the evidence shows
Helps AI engines discover your best pages	Ahrefs: 97% of live files got zero requests in May 2026
Improves how often you get cited in AI answers	SE Ranking/SEJ: no significant correlation, ~300,000 domains, November 2025
Is required for AI crawlers to read your site at all	Not required; crawlers already read server-rendered HTML with or without it
Is useful for steering coding agents away from deprecated code	Yes, documented; Stripe uses it exactly this way
Replaces or overlaps with sitemap.xml and robots.txt	No; different files, different audiences, different purposes

Who is actually using it, and for what

The real adopters are running developer and agent tooling, not chasing search visibility. Cloudflare began auto-generating llms.txt and llms-full.txt for its customers on 26 September 2025, framed as a machine-readable map to help models use site content at inference time, a hosting-layer convenience rather than an SEO claim.

Stripe added an llms.txt in March 2025 with an instructions section, analysed by Apideck in February 2026, that steers coding agents away from deprecated API methods and toward the current ones. That is documentation control aimed at Claude Code, Cursor, and similar tools writing integration code against Stripe’s API. It has nothing to do with search rankings, and Stripe never framed it that way.

That is the pattern worth copying: llms.txt earns its keep where an AI agent needs to act correctly against your product, like a coding assistant integrating an API. It has not earned its keep as a discovery or citation mechanism for content marketing.

How we ship it on every build

We generate llms.txt at build time from our own content collections: services, case studies, and blog posts, each pulled fresh through the same data source that renders the pages themselves, so the file can never drift out of sync with what is actually live. It regenerates on every build, dated to that build, listing current prices, case stats, and the most recent articles. We do not ship a separate llms-full.txt; one summary file has been enough for our own catalogue so far.

Our sitemap is generated the same way, through a standard Astro sitemap integration, and our RSS feed is its own build-time route pulling from the same post collection. On every push to main, once the build succeeds, our CI pipeline pings IndexNow so Bing picks up new and changed URLs without waiting for a crawl cycle.

None of that is a bet on llms.txt doing the heavy lifting. It is table stakes plumbing that costs nothing to keep current, built the way we build everything else here: reproducibly, from source, with no manual step to forget.

What to do instead if you want AI engines to cite you

Build pages that answer the query in the first two or three sentences, name a real statistic with its source, and put that answer in the actual page markup rather than in a separate file the crawler has no reason to check. The evidence above points the same direction every time: the file is not where citation gets decided, the page is.

We wrote the fuller mechanics separately in our piece on generative engine optimization: direct, self-contained answers near the top of a section, named statistics, and citations inside the text itself are what the research actually ties to AI-answer visibility. None of it involves a manifest file at the root of your domain. If your team is still stuck getting any of this shipped on a weekly cadence, that operational gap, not a missing llms.txt, is usually what we find first; we covered why in the piece on why most AI pilots stall.

This is also where proof matters more than a promise about a file format. Our own group runs a 363-product catalogue with daily publishing and structured data on every page, and the citation behavior we chase there comes from the writing and the markup on the page itself, not from a manifest nobody’s crawler is reading yet. We build that same discipline into every content engine we ship for a client: fixed price, fixed scope, and the client owns the pipeline once we hand it over.

The plain version

llms.txt costs almost nothing to add and will not hurt you, so add it if you already have the data pipeline to keep it current. Do not budget time or client expectations around it moving a single citation, because the data through June 2026, from two separate studies and Google’s own John Mueller, says it currently does not. Spend that time instead on server-rendered pages that answer the question directly and name their sources, since that is what the evidence actually rewards.

Describe your content operation and you will get a written plan for a citation-ready content engine within one business day. No meetings, fixed price, and you own the pipeline outright. Start async.