Does ChatGPT respect robots.txt? Not the bot that matters
OpenAI documents four bots. Two are bound by robots.txt, one only ever visits ad pages you submitted yourself — and the fourth, the one that actually opens your pages when a ChatGPT user asks a question, carries a documented exemption. And our own logs show the file was never where access got decided anyway.
The short answer. robots.txt demonstrably governs GPTBot (training) and OAI-SearchBot (search); OAI-AdsBot only ever visits ad pages you submitted yourself. It does not reliably govern ChatGPT-User — the fetcher that reads a page live, the moment a user asks about it. OpenAI's own documentation says of it: «robots.txt rules may not apply». We re-read that page on 15 August 2026; the sentence is there.
Who can close the tab. If you neither want AI assistants reading your site nor care whether they recommend you, this changes nothing today. Your firewall is already making the decision for you — probably more aggressively than you think.
Why the rest of you should read on. Because the file everyone edits is not the layer that decides. Our own robots.txt explicitly welcomes every OpenAI bot by name — and in the 23 hours before publishing this, 95% of ChatGPT-User requests to our site were blocked anyway, by two firewall layers we had to discover ourselves.
- The four OpenAI bots and what each one actually does, from the documentation, not from folklore.
- The exact sentence in which OpenAI exempts ChatGPT-User from robots.txt.
- European market numbers: how often AI fetchers reach pages that were explicitly disallowed.
- Our own measured case, in the opposite direction: a wide-open robots.txt and a firewall that blocks 95% regardless.
On 14 August, Search Engine Journal reported on TollBit's «State of the Bots» report for the first half of 2026, and put a sentence from OpenAI's crawler documentation in front of a much larger audience than developer docs usually get. The sentence has been sitting there in plain sight: ChatGPT's live fetcher, the agent that opens your page when a user asks about your business, is described by OpenAI itself as one that robots.txt «may not» bind.
Most of the coverage stops at the outrage. We run the other experiment daily: our site welcomes every AI bot in robots.txt, explicitly, by name — and our firewall data shows the welcome is worth exactly nothing. The two facts belong in the same article, because together they say something neither says alone: robots.txt is not where AI access gets decided — in either direction. The real decision happens in layers most site owners have never looked at.
What ChatGPT-User is
ChatGPT-User is the user agent OpenAI sends when a ChatGPT conversation needs to open a web page in real time — a user asks a question, the assistant fetches your page to answer it, and your server sees a request signed ChatGPT-User/1.0. It is not a crawler that wanders the web on its own schedule; OpenAI states it «is not used for crawling the web in an automatic fashion».
That single sentence of definition carries the whole argument of this article: a fetch triggered by a human question is, in OpenAI's reading, the human's visit, not the bot's — and robots.txt was written for bots.
OpenAI's four bots, in a table
OpenAI's crawler documentation, re-read in full on 15 August 2026, describes four user agents. Confusing them is the single most common mistake we see in conversations about «blocking ChatGPT», because each one answers to a different lever.
| User agent | What it does | Does robots.txt bind it? |
|---|---|---|
| GPTBot | Crawls content that may be used to train OpenAI's foundation models | Yes — disallowing it signals your content should not be used in training |
| OAI-SearchBot | Crawls to surface your site in ChatGPT's search results | Yes — opted-out sites are not shown in ChatGPT search answers, «though can still appear as navigational links» |
| OAI-AdsBot | Visits landing pages submitted as ChatGPT ads, to validate them | The documentation does not invoke robots.txt here — it only visits pages you yourself submitted as ads |
| ChatGPT-User | Opens a page live when a user's question requires it | «robots.txt rules may not apply» — OpenAI's own wording |
Each of the four has a published list of IP ranges — gptbot.json, searchbot.json, adsbot.json, chatgpt-user.json on openai.com — which matters more than it seems: the user-agent string is a plain-text header anyone can fake, and the IP lists are the only way to tell OpenAI's real traffic from impostors wearing its name. We will meet those impostors again in our own data below.
The sentence OpenAI put in writing
Here is the passage, quoted from the documentation, with the two sentences that matter in sequence:
ChatGPT-User is not used for crawling the web in an automatic fashion. Because these actions are initiated by a user, robots.txt rules may not apply.
Note what this is and what it is not. It is not phrased as an announcement — it is the documentation's standing description of how the system works, which the TollBit report and the trade press have now made unavoidable. It is also not a confession of rule-breaking: robots.txt has been a voluntary convention since 1994, and OpenAI is saying openly that for user-initiated fetches it does not consider the convention to apply. You may find the reasoning fair — a person asking about your business is not a crawler — or self-serving. Either way, the practical consequence is identical: a Disallow line for ChatGPT-User is a request, and the documentation tells you the request may be declined.
Two smaller details from the same page deserve more attention than they get. First, robots.txt changes are not instant even for the bots that do obey them: OpenAI notes it can take «~24 hours from a site's robots.txt update for our systems to adjust». Second, if you allow both GPTBot and OAI-SearchBot, OpenAI «may use the results from just one crawl for both use cases» — one visit, two purposes, which is worth knowing when you read your own logs and wonder why the counts look low.
What Europe's server logs show
The TollBit report gives the market-level picture. According to the figures as reported by Search Engine Journal on 14 August 2026 — we have read SEJ's account, not the primary report — on the European sites in the report, about 15% of the AI page-fetchers identified reached URLs that robots.txt had marked as disallowed in the first half of 2026. For three specific agents — ChatGPT-User, ByteDance's Bytespider and You.com's Youbot — fetches reached disallowed pages on roughly half of the European sites that explicitly listed them.
The same report carries a quieter, stranger number: Europe barely uses the file at all for the newer agents. Only 9% of European websites disallow Claude-User, against 26% in North America; for Perplexity-User the split is 13% against 26%. Whatever you think that gap means — less concern, less awareness, or simply less time spent on a file that demonstrably is not the real lever — it tells you that the robots.txt debate is being conducted mostly on another continent.
Numbers like «15% of fetchers reached disallowed URLs anyway» invite one conclusion: the file under-blocks. Our own data adds the missing half of the picture, because on our site the file points the other way — and reality still ignores it.
The direction nobody checks: open file, closed firewall
The robots.txt on cittago.com is about as welcoming as the format allows. It opens with a comment saying AI and answer engines are wanted, allows everything for everyone except two technical paths, and then names GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, PerplexityBot and others with an explicit Allow: / each. If robots.txt decided access, every OpenAI fetch would be served.
Here is what actually happened, measured on our Cloudflare zone across three dated 23-hour windows. The count is by user-agent string; «blocked» means the request received a 403 from either of two layers — Cloudflare's firewall at the edge, or the hosting provider's security layer (Imunify360) at the origin server.
| Window (23h) | ChatGPT-User requests | Blocked at Cloudflare | Blocked at origin | Served content | Blocked |
|---|---|---|---|---|---|
| 5 August 2026 — before we knew | 191 | 157 | 18 | 16 | 92% |
| 5–6 August — the day spanning our Cloudflare fix | 238 | 81 | 126 | 30 | 87% |
| 14–15 August 2026 — current | 645 | 165 | 450 | 30 | 95% |
The chronology matters. On 5 August we audited the zone and found the first wall: a Cloudflare rule blocking whole continents, written years earlier against spam, which happened to catch the Singapore-heavy infrastructure AI fetchers ride on. We fixed that layer the same day, with an allow rule that exempts requests Cloudflare verifies as genuine bots. The category «blocked at the edge, never reached the server» emptied within hours — for verified traffic. What stayed, and what dominates the current row, is the second wall: the origin server's own security layer, which only the hosting provider can lift — and that has not happened yet. Ten days after we opened the first gate, 450 of 645 requests carrying the ChatGPT-User signature still die at the second one.
Three footnotes on honest reading. The 165 still stopped at Cloudflare are requests that failed bot verification — the allow rule vouches only for traffic Cloudflare can confirm comes from published bot infrastructure, so what remains at that layer is largely impostors wearing the user-agent string. And the control group keeps the experiment honest: in the same current window, Googlebot made 56 requests and bingbot 41, and not one of them was blocked by either layer. Claude-User, for contrast, made 21 requests and all 21 were served — the walls are selective, which is exactly why you cannot deduce them from the outside. And the bookkeeping: a stray response outside the three columns — a 404, say — means a row can miss its total by a request; the 5–6 August row accounts for 237 of its 238 that way.
This inversion is worth stating plainly: TollBit measures sites that said no and got visited anyway. We measure a site that said yes and got blocked anyway. Reading a zone's firewall events against its robots.txt is the first check we run in our SEO for AI search work, because the file records the intention and the log records the truth — and we have yet to audit a site where the two matched on the first look.
We wrote up the original discovery, layer by layer, in our audit of the firewall that was blocking 79% of AI crawlers; the measurement method for who requests which file is the same one we used to check whether anything actually reads llms.txt.
Then and now: where access is decided
| The robots.txt era | August 2026 | |
|---|---|---|
| Who reads your pages | Crawlers on their own schedule | Crawlers, plus live fetchers acting on a user's question |
| Where you express a preference | robots.txt, one file, public | robots.txt for crawlers; WAF and origin firewall rules for everything |
| Who enforces it | The bot, voluntarily | Your CDN and your server, involuntarily — including against traffic you wanted |
| How you verify | You mostly could not | Firewall event logs, per user agent, per response status |
What you need to see your own AI traffic
Not a tool purchase — an hour, and access you already have.
- Your CDN or server logs, filtered by user agent, with the response status next to each request. On Cloudflare the free plan keeps 24 hours of this — enough for a snapshot, if you look on purpose.
- A list of the agents that matter to you. Start with the four OpenAI ones plus Claude-User, PerplexityBot and Googlebot as control.
- A look at both layers. An edge status of 403 with no origin response means your CDN blocked it; a 403 that did reach the origin means your server or its security software did. The fix lives in different dashboards.
- OpenAI's published IP lists, if you want to separate real fetchers from impostors before you draw conclusions.
What this article does not claim
- It does not claim OpenAI ignores robots.txt across the board. GPTBot and OAI-SearchBot are documented as following it, and our data is not evidence either way on that point.
- It does not claim your site looks like ours. One small site, one zone, two specific firewall layers. Your walls will be different — the point is that you have walls you have not read.
- Our counts are by user-agent string. Some of the 645 requests were certainly impostors; the 165 edge-blocked ones largely were. The 450 origin blocks hit verified and fake traffic alike, which is precisely the problem.
- The European figures are TollBit's, read through SEJ's report. We have not audited the primary dataset, and we quote the numbers with that chain attached.
Where you stand, in three thresholds
Threshold one — you have never looked. You have a robots.txt somebody once configured and no idea what your firewall does to AI fetchers. One hour with your CDN logs moves you to threshold two. Most sites we have audited were losing traffic they thought they had welcomed.
Threshold two — you have looked, and found walls. Now it is a plumbing job: identify which layer blocks, decide per agent what you actually want, and write rules that verify bots instead of trusting user-agent strings. Budget days, not minutes — ten days on, our own second wall is still standing.
Threshold three — your logs match your intentions. The remaining work is the slow kind: being worth fetching. A fetcher that can reach you is a precondition, not a strategy — what gets you recommended is still content someone would cite.
Questions nobody has actually asked us
The report is a day old and no client has phoned about a fetch bot. These are the questions we would want answered in their place.
Can robots.txt block ChatGPT completely?
No. It can keep GPTBot out of training data and OAI-SearchBot out of search surfacing — those two are documented as obeying it. ChatGPT-User, the live fetcher, carries the documented «may not apply» carve-out.
If you need a hard no, it has to be enforced at the firewall, against OpenAI's published IP ranges.
Does blocking GPTBot remove my site from ChatGPT's answers?
Not by itself. GPTBot governs training; visibility in ChatGPT search runs through OAI-SearchBot. OpenAI documents that sites opted out of OAI-SearchBot stop appearing in search answers «though can still appear as navigational links».
How do I recognise ChatGPT-User in my logs?
The user-agent string contains ChatGPT-User/1.0 and a link to openai.com/bot. Treat the string as a claim, not proof — OpenAI publishes the real IP ranges in chatgpt-user.json, and anything outside them is someone else wearing the name.
Will I see these visits in Google Analytics?
As a rule, no. Analytics depends on JavaScript running in a browser; a fetcher grabbing your HTML does not oblige. The reliable place to look is your CDN or server log, which is exactly why most owners have never seen this traffic.
Our robots.txt allows AI bots. Are we done?
That was our situation on 4 August: a welcoming file and a 92% block rate. The file is the cheap half. The expensive half is reading your firewall layers — ours were a continent rule at the CDN and a security layer at the hosting — and neither shows up in any SEO tool.
Should I block ChatGPT-User then, since it may ignore the file anyway?
Only if you genuinely want AI assistants unable to read you — and then do it properly, with firewall rules on the published IP ranges. For most businesses we work with, the goal is the opposite: they want to be read, quoted and recommended, and the blocks they have are accidents of old security rules.
Do impostor bots really matter, or is that a technicality?
In our current window, 165 of 645 requests carrying the ChatGPT-User signature failed Cloudflare's bot verification — a quarter of the total. If we had counted user-agent strings alone and called it OpenAI traffic, every percentage in this article would be wrong. Verification is not a technicality; it is the difference between measuring and guessing.
Does the ~24-hour robots.txt delay matter in practice?
Once, and then never again: the day you change the file, do not test five minutes later and conclude nothing happened. OpenAI documents that its systems can take about a day to notice. Check the day after, in the logs, not in the assistant.
Is being fetchable enough to get recommended by ChatGPT?
It is the entry ticket, not the prize. A fetcher that gets a 403 can never cite you, so fixing access comes first. After that, what moves recommendations is whether your pages answer the literal question, with numbers and sources a machine can lift cleanly.
What would you measure monthly?
Three rows from your own logs: requests per AI agent, share of those blocked at each layer, and — separately — how often your brand shows up when you ask the assistants ten fixed questions about your market. The last one is manual and slightly tedious, which is why almost nobody has the data. Ask us again in three to six months 😉
Last updated: 15 August 2026. OpenAI's crawler documentation, re-read in full on this date, describes four bots and states that for user-initiated ChatGPT-User fetches «robots.txt rules may not apply». Our own figures come from the Cloudflare GraphQL API for the cittago.com zone, 23-hour windows dated 5, 6 and 15 August 2026, counted by user-agent string. European market figures are TollBit's «State of the Bots» H1 2026, as reported by Search Engine Journal on 14 August. We update this page when any of these change.
