Nine AI crawlers banned in a file we thought we wrote
On 24 August we opened the robots.txt of five domains we run. One of them was serving a Cloudflare-generated block that bans nine AI crawlers by name — sitting directly above a line we had written ourselves, thirteen days later, inviting every crawler in. Nobody had noticed, because nothing anywhere flags it. Cloudflare announced Bot Preference Sync on 21 August, and a harder deadline sits three weeks out: 15 September 2026.
What we found. Five domains we administer, all behind Cloudflare, all opened on the same day. Four files had no trace of Cloudflare in them. The fifth had nine Disallow lines none of us had written, fenced inside a block marked # BEGIN Cloudflare Managed content.
Why it is about to matter more. On 21 August Cloudflare announced Bot Preference Sync, which generates and updates robots.txt from your dashboard settings, on every plan, on by default for new customers. And on 15 September 2026 the defaults themselves change: multi-purpose crawlers get judged on all their behaviours, most restrictive rule wins. Cloudflare names Googlebot, Applebot and BingBot.
What this page will not do. It will not tell you whether to allow AI crawlers. Cloudflare is right that there is no single answer, and a shop and a publisher want opposite things. It will tell you exactly where to look and what the sentences say.
- You can tell in two minutes whether your file is still yours, from one address in a browser.
- You get the deadline sentence verbatim, with the escape clause that sits in the same paragraph.
- You get the four conditions a crawler must meet not to be blocked — and why they tell you nothing about Googlebot yet.
- You get the one precedence rule that decides which of two contradicting groups actually wins.
A robots.txt is the oldest contract on the web and the least reread. You write it once, at launch, and then it becomes furniture. That is precisely what makes it the perfect place for a rule you did not write to appear without anyone noticing.
Five domains, opened on 24 August
The selection rule was simple: projects we run ourselves, across three countries, all sitting behind Cloudflare. For each one we counted three things — the size of the file in bytes, how many times the word Cloudflare appears in it, and how many Disallow lines it contains.
Four of the five had no Cloudflare content at all. The fifth had three mentions, 2,240 bytes across 69 lines, and nine Disallow lines, none of them ours. All nine came from a fenced block naming the agents one by one: Amazonbot, Applebot-Extended, Bytespider, CCBot, ClaudeBot, CloudflareBrowserRenderingCrawler, Google-Extended, GPTBot and meta-externalagent. Above the fence, the Content Signals legal preamble, citing Article 4 of EU Directive 2019/790.
# BEGIN Cloudflare Managed content
User-agent: *
Content-Signal: search=yes,ai-train=no,use=reference
Allow: /
User-agent: Amazonbot
Disallow: /
User-agent: Applebot-Extended
Disallow: /
...
User-agent: GPTBot
Disallow: /
# END Cloudflare Managed Content
Below the fence, in the same file, a comment we had written, dated: “Open to search engines since 2026-08-11.” Then User-agent: * and Allow: /. The opposite intention, in the same file. When the block above it was switched on we cannot say: the date in the comment is ours, not its.
For contrast, cittago.com on the same day: 543 bytes, 27 lines, zero occurrences of the word Cloudflare, and nine AI crawlers explicitly granted Allow: / — GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-Web, PerplexityBot, Google-Extended, Applebot-Extended and CCBot. Nine banned in one file, nine invited in the other, two zones at the same provider, run by the same people.
Honesty requires one caveat, without which the number above is a good story and a weak proof: from outside you cannot tell whether that block was injected by Cloudflare at delivery or pasted into the source file at some point. Settling it takes the dashboard, not a curl. For a reader it changes nothing — the action is identical either way, and it starts with opening the file.
One detail resolves half of it. The marker reads Cloudflare Managed content, not Cloudflare Bot Preference Sync. That is the older name, the one in the documentation — so that zone is exactly the case Cloudflare describes when it writes that existing customers on the legacy feature “will be prompted to review and confirm their preferences to transition to the new Bot Preference Sync upon its upcoming launch.” The prompt is coming. The question is whether anyone reads it or confirms it by reflex.
Which of the two lines wins
The instinctive reading is that the last rule wins, or the first. Neither is the rule. What decides is specificity, and Google's robots.txt documentation is unambiguous about it: “only one group is valid for a particular crawler,” the one “with the most specific user agent that matches the crawler's user agent,” and “user agent specific groups and global groups (*) are not combined.”
Apply that to the file above and the arithmetic is unpleasant. Each of the nine AI agents has its own named group saying Disallow: /. Our own permission was written as a wildcard, User-agent: * followed by Allow: /. A wildcard group never gets consulted for an agent that has a group of its own. So the line we wrote, deliberately, with a date in a comment beside it, was never going to apply to a single one of the nine crawlers it was written for.
Cloudflare's announcement addresses the neighbouring case and not this one. It promises that existing Disallow directives are preserved when the block is prepended. It says nothing at all about existing Allow directives, because from the feature's point of view there is nothing to say: the block is not overwriting them, it is simply outranking them.
What Bot Preference Sync is
Bot Preference Sync is a Cloudflare feature that generates and updates your site's robots.txt from the AI bot preferences you have set at the zone level, across three categories: search, agent and training.
Cloudflare's own wording, from the announcement by Jin-Hee Lee on 21 August 2026: the feature “reflects what you've set in your AI bot configuration by updating corresponding preferences to your robots.txt, and it can be turned on or off at any time.” The reason sits a paragraph later: “instead of a site owner maintaining a separate static file, Cloudflare generates or updates your robots.txt based on your configuration, so what you say to the world and what you enforce at the edge are kept in sync.”
The problem it solves is real, and anyone who has run two layers of protection has met it. Cloudflare describes it plainly: “there are cases in which your robots.txt states that a crawler is Disallowed from accessing your website, while your enforcement rules actually don't block that crawler. When your stated preferences and your enforced rules disagree, some crawlers treat it as a basis to disregard your preferences or try to bypass your enforced rules.”
What lands in your file
Cloudflare publishes the shape of the block, shortened and anonymised in their example:
# BEGIN Cloudflare Bot Preference Sync
User-agent: TrainingBot1
User-agent: TrainingBot2
User-agent: TrainingBot3
User-agent: MixedUseBot-Extended
Disallow: /
...
# END Cloudflare Bot Preference SyncTwo things to keep. The block is fenced, so it is recognisable at a glance. And the list of agents inside it is not fixed: Cloudflare uses the bots it tracks in BotBase “to periodically update the list of bots that is added to robots.txt when you choose to Block or Disallow a given category.” Your file can therefore change because their catalogue changed, not because you did anything.
There is a small, checkable illustration of that. Cloudflare's managed robots.txt documentation, last updated 3 August 2026, prints a sample block with eight agents. The file being served on the day we looked carried nine — the extra one being CloudflareBrowserRenderingCrawler. Same company, two documents, three weeks apart.
In the dashboard the categories are Search, Agent and Training. In the file, the written signals are search, ai-input and ai-train, plus a fourth field, use, with three values: immediate, reference (the default) and full. They do not map one to one — the dashboard's “Agent” is the file's ai-input. If you go looking in the file for the word you ticked in the panel, you will not find it.
The date that matters is 15 September
The 21 August announcement is about a file. The 1 July one, “Your site, your rules: new AI traffic options for all customers,” is about the door. That is where the deadline lives: “on September 15, 2026, we'll be setting new defaults for each of these three classifications. For all new domains onboarding to Cloudflare, the categories of Training and Agent will be blocked by default on the pages that display ads, while Search will remain allowed by default.”
That part is defensible, and Cloudflare explains it in one sentence worth rereading: “an ad is a signal that a website owner meant for a person to land there and see it.” On those pages, human attention is the point, so the bots that consume it without returning anything stay outside.
It is the next sentence that moves ground, and it reaches domains that have been on Cloudflare for years.
Googlebot, Applebot and BingBot, named
In full, because every clause carries weight: “another change that will apply on September 15 is that multi-purpose crawlers (specifically those that combine Search with Training) will be allowed/blocked according to all of their behaviors, in line with our call for transparency for website owners. Since the defaults will be enforced by the most restrictive applicable rules, multi-purpose crawlers such as Googlebot, Applebot, and BingBot will be blocked by customers who have selected to block Training (either through the new options to manage AI traffic, or through the legacy Block AI bots service).”
Three readings, because it is easy to take more out of this than is in it.
One. It does not say Cloudflare blocks Googlebot. It says Googlebot “will be blocked by customers who have selected to block Training.” The actor is you, through a setting of yours. If you never blocked training, the sentence is not about you.
Two. It explicitly includes “the legacy Block AI bots service.” That is the one-click button from a year ago, at the first Content Independence Day. A great many sites pressed it then, when it meant one thing. From 15 September it means another.
Three. There is a way out, in the same paragraph: a site owner who wants to opt out “can easily mark this in their Security settings any time leading up to September 15, which will confirm that they want no changes on Training crawlers that also crawl for Search purposes.” Cloudflare also promises to keep notifying customers as the date approaches.
The 21 August post adds a qualification that cancels none of the above but changes who it hits. For Training, the Disallow option has been redefined: the no-training preference is written into robots.txt “so that cooperating mixed-use crawlers who take the extra Transparency step can still access your content for search indexing,” and “your Search visibility for cooperating crawlers is unaffected.”
Which reduces everything to one question: does Googlebot cooperate?
Four conditions, and a list we could not find
Cloudflare does not leave “cooperating” to anyone's judgement. It writes four requirements that operators of bots doing both Search and Training must meet “in order to not be blocked when Disallow Training is set”:
- the bot must respect, via any mechanism, a no-training preference in robots.txt;
- they give site owners a way to opt out of AI summaries;
- they provide URL-level visibility into which pages were made available for training, as well as metrics on search results, so you can see how your content was used;
- they can show publicly that disallowing training does not hurt your traditional search results.
The third is the hard one: it asks for a URL-level report of what was made available for training. We did not find such a report at any of the three named engines — which is not proof that none exists, only a statement about what we were able to check. The fourth is harder still, because it asks a vendor to publicly demonstrate something only that vendor can demonstrate.
Cloudflare also says where to check: bots meeting the criteria “are tracked publicly in the AI bot transparency section in Cloudflare Radar.” We went looking on 24 August. The page at radar.cloudflare.com/bots, opened in a real browser and read in full, contains no such section and does not contain the word Googlebot anywhere. Four other plausible addresses on the same domain returned 403 to a plain HTTP client, so about those we can say nothing — neither that they exist nor that they do not.
What follows, carefully: not that the list does not exist. Only that, three weeks from the deadline, a site owner who wants to know whether Googlebot stays at their door has nowhere to look. And the decision has to be made before 15 September.
What you need to check today
- A browser. Open
yourdomain.com/robots.txt. It is public by definition, to you as to anyone. - One word to search the page for: Cloudflare. If it appears, the file is not written entirely by you.
- Access to the zone's Cloudflare dashboard, at the AI bot settings. That is where Search, Agent and Training are set — and the opt-out from the new defaults lives one section over, in Security settings.
- Five minutes and somebody who knows what the site wants. The technical part is a checkbox. The hard part is deciding whether you want to appear in AI assistants' answers, and that is not an infrastructure question.
If you do not know who configured your zone or what is ticked inside it, that is exactly the sort of thing we open when we handle Cloudflare setup and configuration for a website — an hour, and you find out what your own file says about you.
What it does not do
- It blocks nothing. Cloudflare's documentation is blunt: “robots.txt compliance is voluntary. The file expresses your preferences, but it does not prevent crawlers from accessing your content at a technical level. Some crawler operators may disregard your robots.txt directives and crawl your content regardless.” The same page names the tool for actual blocking: “if you want to enforce crawl blocking rather than request it, use AI Crawl Control.” Two different things, in two different places.
- It does not read your custom rules. It is built for category-wide policy rather than case-by-case, and “will not directly read from individual custom rules with more complex logic.” If you have a negotiated exception with one operator, the sync has to come off.
- It does not delete your prohibitions. The block is prepended and existing
Disallowdirectives are kept. On existing permissions, silence — and, as above, silence is enough. - It does not decide for you. Cloudflare is explicit that there is no single right answer: an online shop may want everything crawled and trained on “so its products surface when a shopper asks a chatbot for the best sofa for a small apartment,” while an ad-funded publisher wants the opposite.
- It is not retroactive. Whatever has already been crawled and absorbed into a model stays there. The preference you set works forward, not backward.
Before and now, in a table
| What | Until now | From now, and from 15 September |
|---|---|---|
| Who writes robots.txt | You, once, by hand | The dashboard, whenever the bot catalogue changes |
| AI traffic categories | One button, “Block AI bots” | Three: search, agent, training — plus a use level |
| Multi-purpose crawler | Tracked under one purpose | Tracked under all, most restrictive rule wins |
| New domain with ads | No default blocks | Training and agent blocked on ad-serving pages |
| Verified bot status | Verified meant allowed by default | Verified means allowable within its category |
The words, in a table
| Term | What it means |
|---|---|
| Search | Any behaviour that collects or indexes your content so it can answer questions about it later. Cloudflare writes that owners should expect referral traffic or other equitable compensation in return. |
| Agent | Automated behaviour acting, usually in real time, on a person's behalf. Includes chat fetch bots and browser-use agents. The key: there is a human waiting at the other end. |
| Training | A crawler taking your content to train or fine-tune a model. The key: your data is permanently absorbed into the model's architecture. |
| Mixed-use crawler | Cloudflare's definition: “bots that blend search, agent use, and training behind a single user agent.” |
| Content-Signal | The line written into robots.txt that states the preference, with the fields search, ai-input, ai-train and use. A declared preference, not a block. |
| use | The fourth field, with three values: immediate (interact, but store and reuse nothing), reference (index, excerpt and link back — the default) and full (summarise and reproduce). |
| BotBase | Cloudflare's catalogue of tracked bots. It feeds the agent list written into robots.txt, which is why your file can change because the catalogue changed. |
Where you stand, in three thresholds
First threshold — your site is not behind Cloudflare. None of this applies to you today, and your file is exactly as you left it. The useful thing for you is different: open it anyway, because the odds that it holds a line dropped in four years ago by a plugin are higher than you think.
Second threshold — you are on Cloudflare and you do not know what is ticked. This is the common case and the cheapest to fix: two minutes on the file, five in the dashboard. If you find a # BEGIN Cloudflare fence, read it to the end and compare it with what you thought your site was saying.
Third threshold — you pressed “Block AI bots” at some point, deliberately, and have not revisited it since. You are precisely the person in Cloudflare's sentence about 15 September. That does not mean you must change anything; it means the decision gets made now, knowingly, rather than in three weeks as a side effect.
If you would rather not hold the date in your head, that is what we do when we handle Cloudflare setup for a website: cache rules, security level and now the AI bot policy too, so the file and the edge say the same thing. On the visibility side of the same coin we have written separately, in llms.txt and cats.txt, and on what it looks like when crawlers really are kept out, in AI crawlers blocked by the firewall.
Questions nobody has actually asked us
None of these arrived by email. They are the questions we had to answer ourselves in order to write the article, and the answers are short because the documentation is short.
Does this apply to me if I have never opened Cloudflare's bot settings?
The 15 September change only bites if training is currently blocked on your zone. But that can be true without you having chosen it recently: the one-click Block AI bots button from 2025 counts, and Cloudflare names it explicitly. The only way to know is to look at the zone, not to remember.
Who actually wrote my robots.txt?
Open it and find out. A block generated by Cloudflare is fenced by comment markers that start with # BEGIN Cloudflare, so it is obvious once you look. If that fence is there, at least part of the file was written by a dashboard setting rather than by a person.
What happens when the same user-agent appears twice in one file?
The relevant rule is not about order, it is about specificity: a group that names a crawler by name takes precedence over the wildcard group. In the file we found, each AI agent had its own group with Disallow: /, while the site's own permission was only a wildcard Allow: /. The wildcard never gets a say for those agents. Cloudflare's announcement does not discuss this case at all.
Is robots.txt the same as blocking?
No, and Cloudflare's own documentation says so: “robots.txt compliance is voluntary. The file expresses your preferences, but it does not prevent crawlers from accessing your content at a technical level.” Blocking happens at the edge, as a rule. The whole point of Bot Preference Sync is to stop the two layers from disagreeing.
I run a publisher site. What changes for me?
At onboarding there is now an option worded “I monetize from pages with ads on this domain”, and selecting it sets Training to Disallow by default. From 15 September, new domains get Training and Agent blocked by default on pages that display ads, with Search still allowed. Both settings stay changeable.
Can I keep the old managed robots.txt instead?
Cloudflare says existing customers on the legacy managed robots.txt will be prompted to review and confirm their preferences in order to transition to Bot Preference Sync when it launches. It does not say the legacy behaviour survives that prompt, and it does not give a date for the prompt.
Does any of this touch llms.txt?
No. llms.txt is a separate, unofficial convention that some site owners publish for AI assistants, and Cloudflare's feature does not write it or read it. If you maintain one, it stays exactly as you left it — which also means it can end up contradicting the robots.txt the dashboard now writes.
Will Cloudflare warn me before it changes my file?
For the defaults it promises to: “we’ll also continue to notify customers of the upcoming change to defaults as we approach September 15.” For the bot list inside the block, no warning is described — the list is refreshed from BotBase periodically, which means the file can change on its own.
How do I opt out, exactly?
Cloudflare's wording: a site owner who wants to opt out of the new default configurations “can easily mark this in their Security settings any time leading up to September 15, which will confirm that they want no changes on Training crawlers that also crawl for Search purposes.” It is a security setting, not a bot setting, which is a good reason to look for it before the week it matters.
Is 15 September a hard date?
It is the date Cloudflare published, on its own blog, on 1 July 2026, and it has not been withdrawn as of 24 August. Vendors move dates. What does not move is that the decision is cheaper to make now, deliberately, than to discover afterwards from a traffic graph.
Last updated: 24 August 2026. Quotations about Bot Preference Sync come from “Say it once: introducing Bot Preference Sync,” by Jin-Hee Lee, published on the Cloudflare blog on 21 August 2026 and read in full on 24 August 2026. Quotations about the 15 September 2026 defaults, the three category definitions and the content-use levels come from “Your site, your rules: new AI traffic options for all customers,” published on the same blog on 1 July 2026 and read the same day. The sentence on the file being voluntary comes from the product documentation for managed robots.txt. The five-domain audit was run on 24 August 2026 with plain HTTP requests to each site's public /robots.txt; the byte, line and directive counts were taken from the files downloaded at that moment. We did not find the AI bot transparency section on Cloudflare Radar that the 21 August announcement points to: radar.cloudflare.com/bots was opened in a real browser, and four other plausible addresses on the same domain returned 403 to a plain HTTP client. We will update this page if the public list of crawlers meeting the four conditions appears, if Cloudflare publishes an exact general-availability date, or if the 15 September deadline moves.
Sources: Cloudflare Blog — Say it once: introducing Bot Preference Sync (21 August 2026) · Cloudflare Blog — Your site, your rules: new AI traffic options for all customers (1 July 2026) · Cloudflare Docs — Managed robots.txt · Cloudflare Changelog — New options to manage AI traffic (1 July 2026) · Google Search Central — robots.txt specification
