# Does ChatGPT respect robots.txt? Not the bot that matters

> OpenAI documents 4 bots. ChatGPT-User — the one reading pages for users — may ignore robots.txt. What actually decides access is the firewall: our data.

Source: https://cittago.com/blog/chatgpt-user-robots-txt-2026/  
Publisher: Cittago — a digital studio in Cluj-Napoca, est. 2011  
Published: 2026-08-15  
Language: en

---

OpenAI documents four bots. Two are bound by robots.txt, one only ever visits ad pages you submitted yourself — and the fourth, the one that actually opens your pages when a ChatGPT user asks a question, carries a documented exemption. And our own logs show the file was never where access got decided anyway.

**The short answer.** robots.txt demonstrably governs GPTBot (training) and OAI-SearchBot (search); OAI-AdsBot only ever visits ad pages you submitted yourself. It does not reliably govern ChatGPT-User — the fetcher that reads a page live, the moment a user asks about it. OpenAI's own documentation says of it: *«robots.txt rules may not apply»*. We re-read that page on 15 August 2026; the sentence is there.

**Who can close the tab.** If you neither want AI assistants reading your site nor care whether they recommend you, this changes nothing today. Your firewall is already making the decision for you — probably more aggressively than you think.

**Why the rest of you should read on.** Because the file everyone edits is not the layer that decides. Our own robots.txt explicitly welcomes every OpenAI bot by name — and in the 23 hours before publishing this, 95% of ChatGPT-User requests to our site were blocked anyway, by two firewall layers we had to discover ourselves.

- The four OpenAI bots and what each one actually does, from the documentation, not from folklore.
- The exact sentence in which OpenAI exempts ChatGPT-User from robots.txt.
- European market numbers: how often AI fetchers reach pages that were explicitly disallowed.
- Our own measured case, in the opposite direction: a wide-open robots.txt and a firewall that blocks 95% regardless.

On this page

1. [What ChatGPT-User is](#what-is)
2. [OpenAI's four bots, in a table](#four-bots)
3. [The sentence OpenAI put in writing](#the-sentence)
4. [What Europe's server logs show](#europe)
5. [The direction nobody checks: open file, closed firewall](#our-case)
6. [Then and now: where access is decided](#then-now)
7. [What you need to see your own AI traffic](#need)
8. [What this article does not claim](#not-say)
9. [Where you stand, in three thresholds](#thresholds)
10. [Questions nobody has actually asked us](#faq)

On 14 August, Search Engine Journal reported on TollBit's «State of the Bots» report for the first half of 2026, and put a sentence from OpenAI's crawler documentation in front of a much larger audience than developer docs usually get. The sentence has been sitting there in plain sight: ChatGPT's live fetcher, the agent that opens your page when a user asks about your business, is described by OpenAI itself as one that robots.txt «may not» bind.

Most of the coverage stops at the outrage. We run the other experiment daily: our site *welcomes* every AI bot in robots.txt, explicitly, by name — and our firewall data shows the welcome is worth exactly nothing. The two facts belong in the same article, because together they say something neither says alone: **robots.txt is not where AI access gets decided — in either direction.** The real decision happens in layers most site owners have never looked at.

## What ChatGPT-User is

ChatGPT-User is the user agent OpenAI sends when a ChatGPT conversation needs to open a web page in real time — a user asks a question, the assistant fetches your page to answer it, and your server sees a request signed ChatGPT-User/1.0. It is not a crawler that wanders the web on its own schedule; OpenAI states it «is not used for crawling the web in an automatic fashion».

That single sentence of definition carries the whole argument of this article: a fetch triggered by a human question is, in OpenAI's reading, the human's visit, not the bot's — and robots.txt was written for bots.

## OpenAI's four bots, in a table

OpenAI's crawler documentation, re-read in full on 15 August 2026, describes four user agents. Confusing them is the single most common mistake we see in conversations about «blocking ChatGPT», because each one answers to a different lever.

Each of the four has a published list of IP ranges — gptbot.json, searchbot.json, adsbot.json, chatgpt-user.json on openai.com — which matters more than it seems: the user-agent string is a plain-text header anyone can fake, and the IP lists are the only way to tell OpenAI's real traffic from impostors wearing its name. We will meet those impostors again in our own data below.

## The sentence OpenAI put in writing

Here is the passage, quoted from the documentation, with the two sentences that matter in sequence:

> ChatGPT-User is not used for crawling the web in an automatic fashion. Because these actions are initiated by a user, robots.txt rules may not apply.

Note what this is and what it is not. It is not phrased as an announcement — it is the documentation's standing description of how the system works, which the TollBit report and the trade press have now made unavoidable. It is also not a confession of rule-breaking: robots.txt has been a voluntary convention since 1994, and OpenAI is saying openly that for user-initiated fetches it does not consider the convention to apply. You may find the reasoning fair — a person asking about your business is not a crawler — or self-serving. Either way, the practical consequence is identical: **a Disallow line for ChatGPT-User is a request, and the documentation tells you the request may be declined.**

Two smaller details from the same page deserve more attention than they get. First, robots.txt changes are not instant even for the bots that do obey them: OpenAI notes it can take «~24 hours from a site's robots.txt update for our systems to adjust». Second, if you allow both GPTBot and OAI-SearchBot, OpenAI «may use the results from just one crawl for both use cases» — one visit, two purposes, which is worth knowing when you read your own logs and wonder why the counts look low.

## What Europe's server logs show

The TollBit report gives the market-level picture. According to the figures as reported by Search Engine Journal on 14 August 2026 — we have read SEJ's account, not the primary report — on the European sites in the report, **about 15% of the AI page-fetchers identified reached URLs that robots.txt had marked as disallowed** in the first half of 2026. For three specific agents — ChatGPT-User, ByteDance's Bytespider and You.com's Youbot — fetches reached disallowed pages on roughly **half of the European sites that explicitly listed them**.

The same report carries a quieter, stranger number: Europe barely uses the file at all for the newer agents. Only **9%** of European websites disallow Claude-User, against 26% in North America; for Perplexity-User the split is 13% against 26%. Whatever you think that gap means — less concern, less awareness, or simply less time spent on a file that demonstrably is not the real lever — it tells you that the robots.txt debate is being conducted mostly on another continent.

Numbers like «15% of fetchers reached disallowed URLs anyway» invite one conclusion: the file under-blocks. Our own data adds the missing half of the picture, because on our site the file points the other way — and reality still ignores it.

## The direction nobody checks: open file, closed firewall

The robots.txt on cittago.com is about as welcoming as the format allows. It opens with a comment saying AI and answer engines are wanted, allows everything for everyone except two technical paths, and then names GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, PerplexityBot and others with an explicit Allow: / each. If robots.txt decided access, every OpenAI fetch would be served.

Here is what actually happened, measured on our Cloudflare zone across three dated 23-hour windows. The count is by user-agent string; «blocked» means the request received a 403 from either of two layers — Cloudflare's firewall at the edge, or the hosting provider's security layer (Imunify360) at the origin server.

The chronology matters. On 5 August we audited the zone and found the first wall: a Cloudflare rule blocking whole continents, written years earlier against spam, which happened to catch the Singapore-heavy infrastructure AI fetchers ride on. We fixed that layer the same day, with an allow rule that exempts requests Cloudflare verifies as genuine bots. The category «blocked at the edge, never reached the server» emptied within hours — for verified traffic. What stayed, and what dominates the current row, is the **second wall**: the origin server's own security layer, which only the hosting provider can lift — and that has not happened yet. Ten days after we opened the first gate, 450 of 645 requests carrying the ChatGPT-User signature still die at the second one.

Three footnotes on honest reading. The 165 still stopped at Cloudflare are requests that failed bot verification — the allow rule vouches only for traffic Cloudflare can confirm comes from published bot infrastructure, so what remains at that layer is largely impostors wearing the user-agent string. And the control group keeps the experiment honest: in the same current window, Googlebot made 56 requests and bingbot 41, and **not one of them was blocked by either layer**. Claude-User, for contrast, made 21 requests and all 21 were served — the walls are selective, which is exactly why you cannot deduce them from the outside. And the bookkeeping: a stray response outside the three columns — a 404, say — means a row can miss its total by a request; the 5–6 August row accounts for 237 of its 238 that way.

This inversion is worth stating plainly: TollBit measures sites that said *no* and got visited anyway. We measure a site that said *yes* and got blocked anyway. Reading a zone's firewall events against its robots.txt is the first check we run in our [SEO for AI search](https://cittago.com/services/seo-ai-search/) work, because the file records the intention and the log records the truth — and we have yet to audit a site where the two matched on the first look.

We wrote up the original discovery, layer by layer, in [our audit of the firewall that was blocking 79% of AI crawlers](https://cittago.com/blog/ai-crawlers-blocked-firewall-2026/); the measurement method for who requests which file is the same one we used to check [whether anything actually reads llms.txt](https://cittago.com/blog/llms-txt-cats-txt-2026/).

## Then and now: where access is decided

## What you need to see your own AI traffic

Not a tool purchase — an hour, and access you already have.

- **Your CDN or server logs**, filtered by user agent, with the response status next to each request. On Cloudflare the free plan keeps 24 hours of this — enough for a snapshot, if you look on purpose.
- **A list of the agents that matter to you.** Start with the four OpenAI ones plus Claude-User, PerplexityBot and Googlebot as control.
- **A look at both layers.** An edge status of 403 with no origin response means your CDN blocked it; a 403 that did reach the origin means your server or its security software did. The fix lives in different dashboards.
- **OpenAI's published IP lists**, if you want to separate real fetchers from impostors before you draw conclusions.

## What this article does not claim

- **It does not claim OpenAI ignores robots.txt across the board.** GPTBot and OAI-SearchBot are documented as following it, and our data is not evidence either way on that point.
- **It does not claim your site looks like ours.** One small site, one zone, two specific firewall layers. Your walls will be different — the point is that you have walls you have not read.
- **Our counts are by user-agent string.** Some of the 645 requests were certainly impostors; the 165 edge-blocked ones largely were. The 450 origin blocks hit verified and fake traffic alike, which is precisely the problem.
- **The European figures are TollBit's, read through SEJ's report.** We have not audited the primary dataset, and we quote the numbers with that chain attached.

## Where you stand, in three thresholds

**Threshold one — you have never looked.** You have a robots.txt somebody once configured and no idea what your firewall does to AI fetchers. One hour with your CDN logs moves you to threshold two. Most sites we have audited were losing traffic they thought they had welcomed.

**Threshold two — you have looked, and found walls.** Now it is a plumbing job: identify which layer blocks, decide per agent what you actually want, and write rules that verify bots instead of trusting user-agent strings. Budget days, not minutes — ten days on, our own second wall is still standing.

**Threshold three — your logs match your intentions.** The remaining work is the slow kind: being worth fetching. A fetcher that can reach you is a precondition, not a strategy — what gets you recommended is still content someone would cite.

## Questions nobody has actually asked us

The report is a day old and no client has phoned about a fetch bot. These are the questions we would want answered in their place.

Last updated: 15 August 2026. OpenAI's crawler documentation, re-read in full on this date, describes four bots and states that for user-initiated ChatGPT-User fetches «robots.txt rules may not apply». Our own figures come from the Cloudflare GraphQL API for the cittago.com zone, 23-hour windows dated 5, 6 and 15 August 2026, counted by user-agent string. European market figures are TollBit's «State of the Bots» H1 2026, as reported by Search Engine Journal on 14 August. We update this page when any of these change.

| User agent | What it does | Does robots.txt bind it? |
| --- | --- | --- |
| GPTBot | Crawls content that may be used to train OpenAI's foundation models | Yes — disallowing it signals your content should not be used in training |
| OAI-SearchBot | Crawls to surface your site in ChatGPT's search results | Yes — opted-out sites are not shown in ChatGPT search answers, «though can still appear as navigational links» |
| OAI-AdsBot | Visits landing pages submitted as ChatGPT ads, to validate them | The documentation does not invoke robots.txt here — it only visits pages you yourself submitted as ads |
| ChatGPT-User | Opens a page live when a user's question requires it | **«robots.txt rules may not apply»** — OpenAI's own wording |

| Window (23h) | ChatGPT-User requests | Blocked at Cloudflare | Blocked at origin | Served content | Blocked |
| --- | --- | --- | --- | --- | --- |
| 5 August 2026 — before we knew | 191 | 157 | 18 | 16 | **92%** |
| 5–6 August — the day spanning our Cloudflare fix | 238 | 81 | 126 | 30 | **87%** |
| 14–15 August 2026 — current | 645 | 165 | 450 | 30 | **95%** |

|   | The robots.txt era | August 2026 |
| --- | --- | --- |
| Who reads your pages | Crawlers on their own schedule | Crawlers, plus live fetchers acting on a user's question |
| Where you express a preference | robots.txt, one file, public | robots.txt for crawlers; WAF and origin firewall rules for everything |
| Who enforces it | The bot, voluntarily | Your CDN and your server, involuntarily — including against traffic you wanted |
| How you verify | You mostly could not | Firewall event logs, per user agent, per response status |

## Questions and answers

**Can robots.txt block ChatGPT completely?**

No. It can keep GPTBot out of training data and OAI-SearchBot out of ChatGPT search surfacing — both are documented as obeying it. ChatGPT-User, the live fetcher that opens pages when a user asks, carries OpenAI's documented carve-out: because the actions are user-initiated, robots.txt rules may not apply. A hard block has to be enforced at the firewall, against OpenAI's published IP ranges.

**What is the ChatGPT-User user agent?**

ChatGPT-User is the agent OpenAI sends when a ChatGPT conversation opens a web page in real time to answer a user's question. OpenAI states it is not used for crawling the web in an automatic fashion — it acts on demand, which is why OpenAI treats it differently from its crawlers.

**Does blocking GPTBot remove my site from ChatGPT's answers?**

Not by itself. GPTBot governs use in training; visibility in ChatGPT search runs through OAI-SearchBot. Sites opted out of OAI-SearchBot stop appearing in ChatGPT search answers, though OpenAI notes they can still appear as navigational links.

**How do I recognise ChatGPT-User in my server logs?**

The user-agent string contains ChatGPT-User/1.0 and a link to openai.com/bot. Treat the string as a claim, not proof: OpenAI publishes the genuine IP ranges in chatgpt-user.json, and requests from outside those ranges are impostors wearing the name. In our own 23-hour window, about a quarter of requests carrying the signature failed bot verification.

**Will I see ChatGPT-User visits in Google Analytics?**

As a rule, no. Analytics depends on JavaScript executing in a browser, and a fetcher reading your HTML does not oblige. The reliable place to look is your CDN or server log with response statuses — which is why most site owners have never seen this traffic.

**My robots.txt already allows AI bots. Is my site readable by them?**

Not necessarily. On our own site the robots.txt welcomes every OpenAI bot by name, yet 95% of ChatGPT-User requests in a 23-hour window on 15 August 2026 were blocked — by a Cloudflare continent rule and by the hosting provider's security layer. The file records intention; only firewall logs record what actually happens.

**How can I block ChatGPT-User if I really want to?**

At the firewall, not in robots.txt: write a rule matching OpenAI's published chatgpt-user.json IP ranges, or challenge unverified bot traffic at your CDN. Blocking the user-agent string alone stops honest impostors and misses nothing else.

**Why does OpenAI say robots.txt may not apply to ChatGPT-User?**

OpenAI's documented reasoning is that these fetches are initiated by a person asking a question, so they are treated as the user's visit rather than autonomous crawling. robots.txt has been a voluntary convention since 1994; OpenAI is stating openly that for this class of traffic it does not consider the convention binding.

**How long does a robots.txt change take to reach OpenAI's crawlers?**

OpenAI documents that it can take roughly 24 hours from a robots.txt update for its systems to adjust. Test the day after the change, in your logs, rather than five minutes later in the assistant.

**Is being fetchable enough to get recommended by ChatGPT?**

It is a precondition, not a strategy. A fetcher that receives a 403 can never quote you, so access comes first. What moves recommendations afterwards is content that answers the literal question with numbers and sources a machine can cite cleanly.

---

Cittago · https://cittago.com · digital marketing, SEO, AI search, Google Ads and web development for small and medium companies in Romania, Italy and the EU.