llms.txt in 2026: I measured who actually reads it

Quick answer

I checked this on 8 September 2026, with a plain HTTP client and no browser. Of 15 documentation sites, 11 serve a real llms.txt. Of the five companies that operate the major AI crawlers, zero say anywhere in their own crawler documentation that their crawler reads yours. Every mention I found is a vendor pointing at its own index, which is a different thing. The file is still worth shipping, just not for the reason the top search results give you: the actual win is the .md sibling it points at, which measured 12.7x to 138x lighter than the same page served as HTML.

The moment

A client asked me to "add the llms.txt thing for AI SEO".

I said yes. Then I realised I could not name a single system that reads it.

So I spent an afternoon fetching things instead of guessing. Everything below is what an anonymous HTTP client sees, because that is what a crawler is.

Two questions, kept apart

Who publishes an llms.txt, and who reads one.

These are not the same question. Conflating them is where most of the advice goes wrong, and the search results for this topic conflate them constantly.

For publishing I fetched /llms.txt on 15 documentation sites. I counted a pass only when three things held at once: HTTP 200, a content type of text/plain or text/markdown, and a first line that is a Markdown H1. The H1 is required by the specification, and that third check earns its keep in a moment.

For reading I fetched the official crawler documentation of the five companies whose crawlers people actually care about, and counted occurrences of llms.txt. I counted robots.txt in the same pass as a control, so that a zero could not just be my own broken fetch.

Finding 1: nearly everyone publishes one

Eleven of fifteen. That is real adoption, and I want to be fair about it before I start deflating things.

Two of the misses are worth a note. Cursor does publish one, at cursor.com/llms.txt, just not on its docs subdomain. If your docs live on a subdomain, "the root of the site" is genuinely ambiguous, and that ambiguity is going to bite people.

Google is the interesting one.

Finding 2: an HTTP 200 is not a file

Before the Google case, the check that saved me.

docs.cursor.com/llms.txt returns HTTP 200. It is not a file.

curl -s -o /dev/null -w '%{http_code} %{content_type} %{size_download}\n' \
  https://docs.cursor.com/llms.txt
# 200 text/html; charset=utf-8 513949

curl -s -o /dev/null -w '%{http_code} %{content_type} %{size_download}\n' \
  https://docs.cursor.com/this-path-does-not-exist-xyz
# 200 text/html; charset=utf-8 513949

Same status, same content type, same byte count, to the byte. It is a single-page app that answers every unknown path with the docs homepage.

A checker that only looks at the status code counts that as a pass. Mine would have scored 12 of 15 instead of 11.

The fix is the bogus-path control in the second command. Ask for something that definitely does not exist. If it looks identical to the thing you are testing, you learned nothing from the 200.

Now Google. ai.google.dev/gemini-api/docs/llms.txt is the exact URL the llms.txt proposal cites as evidence that Gemini publishes one. To an anonymous client it never resolves. It redirects into an OAuth sign-in loop and gives up after ten hops, ending on an accounts.google.com authorize URL. A signed-in human in a browser may well get the file. A crawler will not, and neither will your agent.

Finding 3: nobody documents reading it

This is the half that changed my mind.

The robots.txt column is the control. It fires everywhere, so the zeros are real zeros and not a broken fetch.

Now read the three non-zero hits, because they do not say what the count suggests. OpenAI's crawler page says "For the complete documentation index, see llms.txt". Perplexity's says "Fetch the complete documentation index at: /llms.txt".

Both are the vendor pointing at its own index. That is site furniture on their docs template. Neither is a statement that their crawler fetches yours.

So the honest score on the reading side is zero out of five.

Finding 4: the quote everyone repeats is 15 months old

Search this topic and you will hit the same sentence everywhere. Google's John Mueller, on Bluesky: "FWIW no AI system currently uses llms.txt."

It is a real quote. Search Engine Roundtable reported it on 18 June 2025, from a post dated 17 June 2025.

That is 15 months ago.

I am not going to lean on it, and neither should you. A statement about a fast-moving ecosystem does not stay true by being repeated, and half the posts citing it do not mention its date at all. That is exactly why I went and measured the current state myself rather than quoting it as news.

What I will say is that my measurement today does not contradict it.

Finding 5: much of that adoption is a default, not a decision

Here is the deflating bit, and it comes from the proposal's own author.

The specification page notes that "documentation platforms generate one automatically". Mintlify does exactly that, building llms.txt for every customer site with no action from the customer.

Anthropic, OpenAI, Perplexity and Cursor all run their docs on it.

So a chunk of my eleven is one vendor's default setting, counted several times. Nobody at those companies necessarily decided anything. That does not make the file bad, but it does mean "look how many major AI labs have adopted it" is a much weaker argument than it reads.

The specification is careful here, and its critics often are not. It says the AI labs "publish llms.txt files for their own developer docs". Publish. It does not claim they read yours.

Finding 6: discovery is already fragmenting

Mintlify advertises the file with HTTP response headers on every docs page, so a tool can find it without guessing the path. I checked whether that is real, using Mintlify's own docs as a known-positive control.

It is real, and it is already inconsistent.

Two different rel values for the same job, on day one. That is what an emerging convention looks like from the inside.

One trap worth repeating, because I fell in it. The headers are on the documentation pages, not on llms.txt itself. My first pass checked the file, got a clean zero across twelve sites, and was completely wrong. The control caught it. Run a known-positive before you believe any zero.

Finding 7: the part that is actually worth it

Chrome's Lighthouse now has an llms.txt audit under its agentic browsing checks, which sounds like momentum until you read what the audit does. It flags a page if the server throws an error fetching the file. If the file is simply missing and returns a 404, the audit is marked "Not Applicable", because, in Google's own words, "providing the file is optional at the moment".

It does not reward you for having one. It does not penalise you for skipping it. Chrome's docs call it "an emerging convention". Most of the coverage I read while checking this calls it a standard, proposed or otherwise. Only one of those descriptions is written by the people shipping the audit.

But there is a genuine payoff here, and it is not about search at all.

The links inside a generated llms.txt point at .md versions of each page. Those are the same content without the application shell. I measured four pairs:

Across the four, 3.23 MB of HTML against 159 KB of Markdown. Call it 20x.

That is not an SEO number. That is a context-budget number, and it lands on the agent you are already running. Pulling four vendor doc pages as HTML can eat a meaningful slice of a working session; pulling them as Markdown barely registers. If you have ever watched a coding agent burn its context window on navigation chrome and cookie banners, this is the fix, and it is available today with no standard required.

What I do now

I ship the file. It costs nothing, most docs platforms generate it anyway, and if consumption arrives later I am already there.

I do not report it to clients as an AI-search tactic, because I cannot show them anything that reads it.

What I actually use it for is the thing that works right now: when I point an agent at vendor documentation, I give it the llms.txt to find the right page, then fetch that page with .md on the end. That is a real saving I can measure in the same session, which is more than I can say for the crawler story.

What this does not prove

Absence from documentation is not proof of non-consumption. A crawler could read llms.txt today and simply not have written it down, and vendors do not document everything they parse. What I measured is that nobody says they do, which is a fact about documentation, not about behaviour.

This is also one snapshot, from one machine, on one day, from a single location. Sites vary responses by region and by client. And 15 sites is a sample I picked, not a census.

If you want to check whether any of this still holds, the commands are above. That is rather the point.

Postscript: I did add the file for the client. I just changed what I wrote in the invoice line.