Does llms.txt work? 4 requests in 23 hours, zero AI
We have the file at our root too. This morning I looked at who actually downloads it. Four requests in 23 hours — three of them mine, from a terminal, while writing this. robots.txt, over the same window: 95.
The short answer. Not for the assistants you care about. Google states in its own documentation that Search ignores the file. On our zone, over 23 hours and 19,489 requests, /llms.txt was fetched four times and not once by an AI crawler. In the largest public study — 137,210 domains — 97% of these files received no requests at all in a month.
Who can skip this. If you already have an llms.txt that generates itself and costs you nothing, this will not change your day. Ours is still there. This is for anyone about to pay someone to build one, or to „optimise” it.
Why it is worth reading. Because on 7 August a British consultant published an experiment that dismantles, with a straight face, all four arguments normally used to prove llms.txt works. He invented a fake standard about office cats. It passed every test.
- What Google actually wrote about the file, quoted in full.
- Our complete table: who requests robots.txt, who requests sitemap.xml, who requests llms.txt.
- The 137,210-domain study, and the one row in it nobody quotes.
- The command that answers the question on your own site, in ten minutes.
In 23 hours, on a site that received 19,489 requests, the file /llms.txt was requested four times. Three of those four came from curl, on our own server, because I was checking the file existed while documenting this article. The fourth came from a mobile browser I cannot identify.
Over the same window, /robots.txt was requested 95 times — and among the programs requesting it were precisely the bots llms.txt is supposed to serve: ClaudeBot, PerplexityBot, Claude-User, xAI-SearchBot, YouBot, ExaSearchBot.
We do AI search optimisation for a living — it is literally half of what we sell. So this article is commercially inconvenient. We are publishing it anyway, because the alternative is letting people pay for a text file.
The short answer, and how we got it
There are three independent pieces of evidence, and they point the same way. In order of weight, not of discovery:
- Google says in its own documentation that Search ignores the file. Not „we have no comment”. Ignores.
- The largest public log study, across 137,210 domains, found 97% of these files received zero requests in a single month.
- Our own zone, measured directly: four requests in 23 hours, none from an AI crawler, while the same crawlers were busy reading robots.txt and sitemap.xml.
What none of those establishes is that the file will never matter. It might. What they establish is that today, for the assistants that compose the answers you want to appear in, it does nothing measurable.
What Google says, in writing
This is the single strongest fact in the whole subject, and it comes from the company with the most to gain from staying vague. In its guide to optimising for generative AI features, Google writes:
You don’t need to create new machine readable files, AI text files, markup, or Markdown to appear in Google Search (including its generative AI capabilities), as Google Search itself doesn’t use them.
And then, naming the file directly: „It’s completely fine if you decide to create and maintain LLMS.txt files (or other similar files) for other services or systems that use these files. Doing so will neither harm nor help your site’s visibility or rankings in Google Search, as Google Search ignores them.”
Note what that does and does not settle. It settles Google Search, including AI Overviews and AI Mode. It says nothing about OpenAI, Anthropic or Perplexity. We looked for an equivalent statement on their domains — a published commitment that the assistant reads the file at the moment it composes an answer — and did not find one. Not a denial. An absence.
Earlier, Google's John Mueller had put it less formally on Reddit, in a comment reported on 2 June 2026: „I don’t think anyone knows – it’s purely speculative for now (the file has existed for years, yet none of the AI systems use it — what does it mean?).” The documentation has since caught up with him.
His other line in that exchange is the most practical sentence in the entire topic: „all of this assumes that the most basic agentic optimization is in place, namely: don't block agents.” A site that blocks AI crawlers at the firewall and keeps an immaculate llms.txt has solved the wrong problem. We were in exactly that position until a few days ago, and we published the numbers.
What llms.txt is
llms.txt is a text file proposed in 2024, placed at the root of a website, listing its important pages in a simplified form, on the idea that language models would read it instead of navigating the whole site.
It is not a standard anyone adopted, and there is no body maintaining it. It is a proposal that caught on in the marketing community because it sounds logical and is easy to implement — two qualities that together produce the most durable myths.
Ours looks like a list of links with a one-line description each, regenerated automatically on every publish. It is 12,583 bytes. It has cost us nothing beyond the code that writes it.
What we measured, in 23 hours
The numbers come from Cloudflare's GraphQL API, which holds the request log for our zone. The window is 23 hours because retention on the free plan is 24 and we did not want to touch the edge. Requests were grouped by path and by user-agent, with no filtering applied in advance.
| File | Requests | Served | Who asked |
|---|---|---|---|
| /robots.txt | 95 | 72 | ClaudeBot 11, YandexBot 10, MJ12bot 9, facebookexternalhit 7, PerplexityBot 4, Bytespider 4, Googlebot 3, AhrefsBot 3, SemrushBot 3, xAI-SearchBot 1, YouBot 1, ExaSearchBot 1, Claude-User 1, bingbot 1, rest browsers and scanners |
| /sitemap.xml | 26 | 18 | ClaudeBot 11, AhrefsBot 7, Googlebot 2, bingbot 2, GPTBot 1 (refused), rest browsers |
| /llms.txt | 4 | 4 | curl/8.14.1 3 (us, from our own server), one unidentifiable mobile browser 1 |
| /llms-full.txt | 0 | 0 | nobody |
Source: Cloudflare GraphQL, httpRequestsAdaptiveGroups, cittago.com zone, window 8 August 2026 08:13 UTC to 9 August 2026 07:13 UTC. Total requests to the zone in that window: 19,489. „Served” = a 200, 301, 302 or 304 response.
The curl row deserves its own sentence, because it is the most exposed thing in this article: three of the four requests to our llms.txt in the last 23 hours were made by me, at a keyboard, checking the file was where it should be before writing about it. Without that, the table would show a single request.
The same bots, the same day, on two different files
ClaudeBot requested robots.txt 11 times and sitemap.xml 11 times. It never requested llms.txt. PerplexityBot requested robots.txt four times and llms.txt zero times. GPTBot tried sitemap.xml once and did not ask for llms.txt.
sitemap.xml · ClaudeBot 11 · Googlebot 2 · bingbot 2 · GPTBot 1
llms.txt · AI crawlers 0
That is the point. Not that AI crawlers ignore us — they read us, and it shows. It is that they read robots.txt and sitemap.xml, the two files that have been actual standards for thirty and twenty-five years respectively. The file proposed specifically for them does not appear in the log.
One detail matters for anyone who read our piece on AI crawlers blocked at the firewall: until 5 August, 79% of AI crawler requests were stopped before they reached our site. This measurement was taken after that repair, with the doors open. So this is not a case of nobody being able to reach us. They reach us. They ask for other files.
The 137,210-domain study
Our measurement is small. There is a much larger one, using the same method — server logs, not statements.
Ahrefs published on 15 June 2026 an analysis of 137,210 domains. 28% of them (38,360) had a valid llms.txt, and 97% of those files received no requests at all during May 2026. The roughly 1,100 domains that did get traffic collected about 22,000 requests between them.
The interesting part is not the percentage. It is who asks for the file when anybody does. Ahrefs split the requests across twelve bot categories:
| Who requests llms.txt | Share of requests |
|---|---|
| SEO audit tools | 21.7% |
| Unidentified | 14.9% |
| General web crawlers | 13.1% |
| Tech profiling tools | 11.6% |
| AI agents and infrastructure | 10.5% |
| GEO/AEO audit tools | 5.8% |
| AI training crawlers | 5.3% |
| Dedicated llms.txt scanners | 3.6% |
| Research bots | 2.7% |
| AI assistants | 2.5% |
| AI retrieval bots | 1.1% |
Source: Ahrefs, „We Analyzed 137K Sites: 97% of llms.txt Files Never Get Read”, published 15 June 2026 by Louise Linehan and Xibeijia Guan. The AI categories together account for 19.5%. The remainder to 100% is human and social bots.
Read that table from the top and you find something no GEO deck mentions: the largest single category downloading llms.txt is SEO audit tools, at 21.7%. And the category called „AI assistants” — the programs that actually compose the answer you want to appear in — is 2.5%. More than 12% of requests come from the industry checking the phenomenon: GEO audit tools, dedicated llms.txt scanners and research bots. A good share of the activity around the file is the industry looking at itself.
A second study, smaller but pointing the same way: EZY Research tracked 83 connected sites for twelve weeks, 27 April to 19 July 2026, and counted GPTBot at 3,990 robots.txt fetches against 7 for llms.txt; ClaudeBot 3,120 against 9; PerplexityBot 775 against 0; Googlebot 5,125 against 67. The row worth including because it contradicts us: Meta-ExternalAgent asked for llms.txt more often than robots.txt — 193 against 172. At least one agent does read it.
Cats.txt: the invented standard that passes every test
Cats.txt is a fake standard published in August 2026, in which a British SEO consultant declared a company's office cats — and showed that it passes exactly the same four tests used as proof that llms.txt works.
On 7 August 2026, Search Engine Journal reported the experiment by Mark Williams-Cook, a director at the UK agency Candour, with more than twenty years in search and the founder of SEO tools including AlsoAsked. We read both the SEJ write-up and his own post, which is where the quotes below come from.
The file declares a firm's cats: names, job titles, breed, and an affection metric called „PurrLevel”, from 0 to 10. He wrote a serious-looking specification and announced it on LinkedIn as „the missing standard for SEO and GEO”. Another technical specialist, Dave Smart, adopted it on his own site — „an early adopter of a standard I had built to be nonsense”, as Williams-Cook puts it.
Then the results. Crawlers downloaded the file — PerplexityBot, GPTBot, ClaudeBot, Googlebot „and a supporting cast of lesser crawlers”. Google indexed it and, in his words, „will even offer to let you claim it in Search Console and ‘get indexing and ranking data,’ with the straightest of faces”. ChatGPT returned, as fact, details that existed only in that file. And roughly two weeks after launch, asked whether cats.txt helps, ChatGPT replied:
Yes — cats.txt can potentially help you rank in both search engines and LLM-driven systems.
The four „proofs” and what they actually prove
| The „proof” | What it actually demonstrates |
|---|---|
| „The bots fetched it” | That you put a file on the internet. Crawlers request files from domain roots out of habit, not out of allegiance to a standard |
| „Google indexed it” | That it is a public text document. Google indexes text files; that says nothing about their effect |
| „ChatGPT cited something only in it” | That the model found the page through search, like any other page. Not that it used it as a site map |
| „ChatGPT said it helps” | That enough people wrote on the internet that it helps. A model does not evaluate; it averages what it read |
Restated from Mark Williams-Cook's cats.txt experiment, published on his own blog and reported by Search Engine Journal on 7 August 2026.
The first column is the test most case studies in this field apply. The second is what survives once the same test passes for a file about cats.
You will also find the same chain presented as demonstration in other languages: a file goes up at the root, and within days new queries appear in Search Console, therefore the file worked. Those queries are real — we have them too, and we published them: the searches written by AI assistants, 42 out of 223 over 28 days. But they appear because an assistant is searching your topic, not because it read your file. To turn coincidence into cause you would need a measurement without the file, over the same period, on the same site. Nobody runs it, because it is inconvenient.
The convergence problem: why ChatGPT agrees with you
The most useful part of the cats.txt experiment is not the joke, it is the explanation.
When you ask a model whether llms.txt helps, it does not run an experiment. It cannot. What it does is return the average shape of what it has read on the subject — thousands of blog posts and sales decks written by people who already assume the answer is yes. The model reflects the consensus back at you, confidently.
The proof that this is exactly what happens: once the internet worked out that cats.txt was a joke, the models' answers changed and began correctly identifying it as satire. Not because they reasoned better. Because the surrounding text changed.
The consequence for anyone who checks their ideas by asking an assistant: a model confirms what the internet believes, not what is true. On subjects where the internet is right, that is a good tool. On subjects where an industry has talked itself into something, it hands the self-persuasion back to you, nicely phrased.
What this data does not say
| What someone might read into it | What the data actually says |
|---|---|
| „llms.txt will never matter” | Unknowable. We measure the present: in 23 hours, on our zone, no AI crawler asked for it |
| „No AI tool reads it” | False. Meta-ExternalAgent asked for it more often than robots.txt in the EZY study, and 19.5% of requests in the Ahrefs data come from AI categories. Agent frameworks and MCP integrations do use it |
| „Your measurement is conclusive” | It is a 23-hour window on a small site. Its value is that it can be repeated, not that it is final |
| „Delete llms.txt from your site” | We did not delete ours and we do not recommend it. It costs nothing and breaks nothing. The argument is about what you pay for it |
| „Google said nobody uses it” | Google said Google Search ignores it. That is not the same statement, and it does not cover other vendors |
Why we keep ours anyway
Because it costs zero. It regenerates itself on every publish, from the same content that generates the sitemap. Nothing to maintain, nothing to pay, no time consumed.
If a system starts reading it tomorrow, we have it. If not, we have spent twelve kilobytes. That is the only cost-benefit ratio in which an unproven thing is worth keeping: when the cost is literally nothing.
What we would never do is sell it as a service, put it in a proposal as a line of value, or bill hours for „llms.txt optimisation”. The sentence that sums it up is Williams-Cook's:
Every hour, and every dollar spent implementing llms.txt, or the next GEO ritual, or the one after that, is an hour and a dollar not spent on something you actually know has value.
What is worth doing instead
The right question is not „which file do I add”, it is „why would a model have reason to mention me”. Here is the order we work in, from what changes the most to what changes the least:
- Do not block the agents. Check the log for who actually reaches you and who gets a 403. On our site, 79% of AI crawlers were being stopped without anyone deciding it. Cheapest repair with the largest effect, and almost nobody checks it.
- Be mentioned elsewhere. Models assemble answers from sources they treat as trustworthy. Most of those are not your website: they are profile directories, comparison sites, trade press, discussions. This is the real lever and it is slow work.
- Answer the literal question literally. If your page does not contain the sentence that answers the question, there is nothing to quote. That has been true of Google featured snippets for a decade and it is true of assistants now. Our step-by-step guide is how to get cited by AI search.
- Clean structure and correct data. Valid markup, headings that say what the page is about, consistent contact details everywhere. Not magic, hygiene.
- A text file at the root. Last on the list, because that is where it belongs.
On the difference between classic optimisation and optimising for assistants, we wrote the first article on this site, on 12 July 2026: GEO versus SEO. The deliberate work on the subject started here on 27 July. We are weeks in, not years, and we say so every time.
How to check it yourself, in ten minutes
You do not need Cloudflare. You need access to your server logs — or to your hosting stats panel, which almost certainly has a requested-files report.
With terminal access on the server, three commands:
# how many times each file was requested, from the access log
grep -c "GET /llms.txt" access.log
grep -c "GET /robots.txt" access.log
# who asked for llms.txt, with the user-agent
grep "GET /llms.txt" access.log | awk -F'"' '{print $6}' | sort | uniq -c | sort -rnOn Cloudflare the same information sits in the GraphQL API, grouping httpRequestsAdaptiveGroups by clientRequestPath and userAgent. The free plan gives you a 24-hour window, which is enough to see the shape.
What you are looking for: the ratio between robots.txt requests and llms.txt requests, and whether any AI crawler name shows up among the latter. If you find something different from what we found, we genuinely want to know — that is the kind of data missing from this entire subject.
The words, in a table
| Term | What it means, briefly |
|---|---|
| llms.txt | Text file proposed in 2024, at a site's root, listing its important pages for language models. Adopted by no search engine |
| robots.txt | Standard file since 1994 telling crawlers what they may access. Everyone reads it |
| Crawler | A program that downloads pages from websites. GPTBot, ClaudeBot, PerplexityBot and Googlebot are crawlers |
| User-agent | The name a program gives when requesting a page. It appears in the server log |
| GEO | Optimising content to be used in the answers of AI assistants. The surfaces are about eighteen months old |
| The convergence problem | A model reflects the consensus of the texts it read, not the result of a test |
Where you stand, in three thresholds
- You have no llms.txt and never thought about it. You have lost nothing measurable. Check instead whether your firewall lets AI crawlers in — that is where real things get lost.
- You have one, generated automatically, costing you nothing. Exactly like us. Leave it and forget it. Do not optimise it, do not extend it, do not give it time.
- Someone is charging you to build or „optimise” it. Ask for the measurement: how many requests the file received, from whom, over what window. No log, no evidence — just a belief.
If you want an answer on your own case, with the log on the table and no magic files, that is what we do at SEO and AI search optimisation. First conversation is free and has no script.
Questions nobody has actually asked us
This was written the day we ran the measurement, so nobody has had time to ask us anything. The questions below are the ones a reader would have, plus the ones that come up anyway in real client conversations whenever the subject turns to AI visibility.
Should I delete llms.txt from my site?
No. It breaks nothing and, if it generates itself, costs nothing. We kept ours.
The only thing we would say is: do not pay for it, and do not list it in a monthly report as an achievement.
So is GEO a con?
No. That a text file does not work does not mean visibility in AI answers is not real. It is real and it is measurable — we track it in Search Console and in the server log.
The con is the idea that it is solved by a technical checkbox. It is solved by what you wrote, where you are mentioned, and whether you let the agents in.
How do I know if I am blocking AI crawlers?
Look in the server log or the firewall panel at the response codes for known user-agents: GPTBot, ClaudeBot, PerplexityBot, ChatGPT-User. If you see 403s, you are blocking them.
On our site 79% of them were being stopped without anyone deciding it explicitly. The whole method is in the article on blocked AI crawlers.
Are the strange Search Console queries caused by llms.txt?
No. They are searches written by AI assistants working on your topic, and they appear whether or not you have the file. We have them too, 42 out of 223 over 28 days.
To demonstrate a link you would need a period without the file, under the same conditions. Nobody publishes that.
Why does Meta read it and the others do not?
We do not know, and we found no public explanation from Meta. The EZY figure is clear — 193 llms.txt requests against 172 for robots.txt — but the reason is not documented.
A reasonable hypothesis is that a recently built agent treats the file as a cheap source of structure. It stays a hypothesis.
Is a 23-hour measurement not too short?
It is, which is why we put the 137,210-domain study and the twelve-week panel next to it. Our window is limited by the 24-hour retention on Cloudflare's free plan.
The value of a short measurement is not that it is final. It is that you can repeat it today, on your own site, in ten minutes.
Would a better-written file change anything?
For the contents of a file to matter, somebody has to request it first. In the window we measured, no AI crawler requested ours, and across 137,210 domains 97% of these files were requested by nobody at all.
So the debate about how to write a better llms.txt is, for now, a debate about the style of a letter nobody opens.
My agency sold me this as a service. What now?
Not a scandal — until a few months ago plenty of serious people thought it was worth doing. The useful question is what happens next, not what happened.
Ask for the report with three numbers: how many requests the file received, from which user-agents, over what period. If they have them, you have an agency that measures. If not, ask what else is being measured.
Does Google's statement settle it for ChatGPT too?
No, and it is worth being precise. Google's documentation covers Google Search, including its generative features. It says nothing about OpenAI, Anthropic or Perplexity.
For those three there is no published commitment either way on their own domains. What we have instead is the log evidence, which points the same direction.
What should I measure monthly instead of files?
Three things, all from sources you control: how many requests you get from AI crawlers and how many get a 403; how many machine-signature queries appear in Search Console and on what topics; and which answers you appear in, checked by hand against ten fixed questions.
Less spectacular than a new file, and the only ones that move. Ask us again in three to six months 😉
Last updated: 9 August 2026. Our own figures come from the Cloudflare GraphQL API for the cittago.com zone, window 8 August 2026 08:13 UTC to 9 August 2026 07:13 UTC, and from the Search Console API for the same property over a 28-day window. Google's position is quoted from its AI optimization guide on developers.google.com, consulted the same day. The 137,210-domain figures come from the Ahrefs study published 15 June 2026; the 83-site panel from EZY Research, published 27 July 2026. The cats.txt experiment was reported by Search Engine Journal on 7 August 2026 and described by its author on his own blog. We will repeat the measurement and update this page if the shape changes.
