Why ChatGPT can't read your site: 79% of AI bots blocked
Our robots.txt welcomed every AI crawler by name. Our firewall returned 403 to four out of five of them. Googlebot, same site and same day, got through 99% of the time. Here is what we found in our own logs, and what a fix actually moved.
What happened. We pulled 23 hours of our own traffic logs to see how many AI crawler requests actually reach cittago.com. Fewer than one in five did. The rest got a 403.
Who this is not for. If nothing sits in front of your site — no CDN, no firewall, no security layer on the server — you probably do not have this problem. If something does, and for most business sites something does, half an hour is worth spending.
Why it is worth reading. Nobody ever ticked a box marked "block AI". The blockade emerged from two reasonable security decisions, made separately, years apart. That is exactly why it is hard to find.
- The one number you can check on your own site tonight, and where it lives.
- A refusal that is served with HTTP 200 and counts as a success everywhere except in the bytes.
- Why your AI crawler dashboard can show green while nothing gets through.
- What one line of firewall rule changed, measured 15 hours later.
Start with the number, because everything else follows from it. Over 23 hours, AI crawlers made 910 requests to our site. 716 of them were refused. In the same window, on the same infrastructure, Googlebot made 79 requests and was refused once.
79% against 1%. Not a policy. An accident.
We put Cloudflare in front of the sites we build and configure it for clients. We have also been writing about AI visibility since our first article on this site went up on 12 July, with the serious, search-and-assistant-oriented work starting on 27 July. Which makes the next sentence the awkward one: for weeks, when we handed an assistant a link to a fresh article of ours, four out of five could not open it. We filed it under "AI being flaky" and moved on. On 5 August we looked at the logs instead of assuming. The flakiness was ours.
| Crawler | Requests | Stopped at the edge | Stopped at the server | Served the page | Refused |
|---|---|---|---|---|---|
| Amazonbot | 228 | 228 | 0 | 0 | 100% |
| ChatGPT-User | 191 | 157 | 18 | 16 | 92% |
| PerplexityBot | 129 | 50 | 0 | 78 | 39% |
| GPTBot | 96 | 53 | 4 | 36 | 59% |
| ClaudeBot | 82 | 0 | 56 | 26 | 68% |
| OAI-SearchBot | 69 | 49 | 8 | 11 | 83% |
| Google-Extended | 58 | 58 | 0 | 0 | 100% |
| All AI crawlers | 910 | 610 | 106 | 185 | 79% |
| Googlebot | 79 | 1 | 0 | 62 | 1% |
| bingbot | 30 | 0 | 0 | 25 | 0% |
Source: Cloudflare GraphQL analytics API, cittago.com zone, 4 August 18:21 to 5 August 17:21 UTC. Crawlers with more than 50 requests, plus the control group; the gap to the total is made up of smaller crawlers. Within each row, what the three columns do not account for are requests that ended in other statuses — redirects and timeouts.
The row that stings is the second one. ChatGPT-User is not a training crawler. It is the live fetcher — the one that opens your page at the moment a real person asks a question your page might answer. Nine out of ten of those were turned away. Everything we have written about getting cited by AI search assumes that request succeeds.
The part most people miss
A refusal that reports itself as a success
A 403 is honest. You can count it, chart it, act on it.
The other kind of refusal is the problem. The server answers 200 OK and hands over a verification screen instead of the page. Every dashboard you own records that as a successful request. The bot asked, the bot received, status 200, everybody happy.
Except the bot received a loading screen.
The only way to see the difference is to stop reading status codes and start counting bytes.
| What the bot receives | Size | HTTP status |
|---|---|---|
| The actual page | 50,000 – 58,000 bytes | 200 |
| Verification screen | ~6,000 bytes | 200 |
| Outright refusal | 1,100 – 1,500 bytes | 403 |
Measured on cittago.com, 5–6 August 2026. The middle row is the one that never shows up as a problem anywhere.
On 6 August, a ChatGPT fetcher coming out of Switzerland requested our Google Ads service page and received 6,064 bytes. The real page is 54,801. Status 200. From the outside, a clean read.
Why every dashboard said green
Before looking at logs, we checked the three places anyone would check. All three were reassuring, and all three were beside the point.
The robots.txt file was immaculate. Explicit permission for GPTBot, ChatGPT-User, ClaudeBot, PerplexityBot, Google-Extended, Applebot-Extended. But robots.txt is a note of good conduct, not a key. It tells a bot what you would like it to do. What happens at the door is decided by the firewall.
The zone settings were mild. Security level essentially off, browser integrity check off, managed WAF off. Nothing aggressive. Read the configuration and you would describe a site that blocks nobody.
The AI crawler control panel showed "allow" for everything. This is the part worth remembering, because Cloudflare documents it themselves: custom rules are evaluated before AI Crawl Control, and what those rules block does not appear in that panel's analytics. So the panel tells the truth about itself and lies about reality. The request was already gone.
There are two doors, not one
The blockade did not come from one place. It came from two, and they had nothing to do with each other.
The first door is the layer in front of your site, which stops requests before they touch your server. The second door is the server itself, with its own security filter, its own rules, usually installed by the hosting provider. Getting past the first one is not the same as getting in.
The proof is the happiest accident in the whole dataset. For unrelated reasons, our zone already carried an exception for exactly one crawler: ClaudeBot. So the data contains a natural experiment — one bot with a pass, everyone else without.
ClaudeBot cleared the first door 82 times out of 82. It was then refused by the server in 56 of those 82 cases.
Fix only the layer in front and you have done half the job with no way of knowing it. Any diagnosis has to separate "stopped at the edge" from "stopped at the server". Two different numbers, two different owners.
AI crawlers come out of Singapore
The rule responsible for the first door was a single line, written years ago against a wave of spam: block traffic from Africa, Asia, Oceania and the Tor network. That is all it said. Nothing about AI. Nothing about bots.
It produced 815 of the 940 blocks recorded on the zone. Eighty-seven percent of everything.
The country breakdown explains the rest: 630 blocks from Singapore, 43 from Australia, 25 from China, 24 from India, 23 from Turkey, 15 from Vietnam. A large share of AI infrastructure egresses through Singapore. A geographic rule written in a different decade of the internet, against a different problem, landed squarely on the one population you now want to let in.
This is not a misconfiguration. It is a rule that aged. The distinction matters, because the second thing happens to everyone.
Prerequisites
Four things you need before you can check
- Access to the panel of whatever sits in front of your site — on Cloudflare, the security section and the list of rules you wrote yourself.
- Analytics that separate edge from origin. Without that split you are guessing which door failed.
- Someone who can change things on the server, or a hosting provider you can open a ticket with. The second door is not in your panel.
- A test page carrying a string nobody could guess. There is a section below on why this is not optional.
The fix: check the credentials, don't open the gate
The obvious move is to add an exception for every AI bot name and be done. It is also the wrong move, for a reason that was playing out in our logs while we worked.
The name a bot declares — its User-Agent — is a text string. Anyone can type it.
During the audit, our logs filled with requests announcing themselves as PerplexityBot, OAI-SearchBot and ChatGPT-User, and going hunting for credentials: /.env, SSH keys, cloud secrets, Kubernetes configs. All refused. None of them was an AI assistant. It was a scanner wearing their names, in real time, in the same window in which we were counting the real ones.
So the rule we wrote does not read names. It requires a cryptographically verified bot — Cloudflare's own confirmation, based on the source address, that the crawler is who it claims to be. The name alone opens nothing.
In practice we widened a rule that already existed rather than stacking a new one on top. It lets verified AI and search crawlers through, and leaves the geographic block fully in place for human traffic and for unverified scrapers. A handful of bots stay out on purpose — the ones that take content and send nothing back.
The change went live on 5 August at 17:33 UTC.
The test
What the AI said against what the logs show
Here is the trap waiting for anyone who tries to verify this themselves: you ask an AI whether it can read your site, and it tells you what it believes happened. It gets this wrong in both directions.
To take belief out of it, we built a test page. It is public, linked from nowhere and absent from the sitemap: cittago.com/ai-access-test/. It carries a code in the first paragraph, a second code at the very bottom, a keyword in the middle and three numbers in a table. The sum of those numbers is written nowhere. It has to be calculated.
No model can return all four unless it actually read the page. The codes are random and the page exists in no index, so they cannot come from memory. Then we read the logs to see what each one received, measured in bytes. Statement and log, side by side.
Perplexity: the right content, without downloading anything
We asked it to read our home page. It returned the exact headline, all five figures from the top section, both listed projects. Correct throughout. It closed with a sentence about how it obtained the content:
"I fetched the page just now."
Cloudflare log, same window: PerplexityBot → 403 on /. 1,133 bytes served. No fetch happened.
The content was right because it was already in their own index, from earlier visits. On a page their index did not have, it reported honestly that it could not retrieve anything.
Gemini: read the page, answered from an older copy
It passed the test — all four values, including the sum it had to work out itself. The log agrees: it received 54,801 bytes, the whole page.
Then, asked about our recent projects, it named one that has not been displayed there for days.
What this means: it read the live page and answered from a copy older than the live page. A successful fetch does not guarantee a current answer.
Access confirmed
Grok: the only clean pass, every time
All four values, including the calculated sum, plus the date printed on the page. Repeated across two different pages on two different days.
Technical note: neither Grok nor Gemini reported the decoy string hidden in the HTML source and invisible when rendered. Both are reading rendered text, not the raw file.
Access confirmed
ChatGPT: two different failures that look identical
On the test page it reported no access. The log shows no request was ever made to that address. It was not blocked: it simply does not fetch a URL it does not already hold in its index, and the page was new and linked from nowhere.
On an indexed page the request did happen — and returned 6,064 bytes instead of 54,801. The verification screen, status 200.
Why the distinction matters: "I can't find the page" and "I was turned away at the door" look the same from outside. Only the log separates them.
Take an AI's word about access to your own site and you will draw the wrong conclusion in both directions. One says it read you without ever asking. Another says it cannot reach you when in fact it never looked.
What the fix moved
We re-ran the same query on 6 August, over the window that begins after the change: 5 August 18:00 UTC to 6 August 08:35 UTC. Roughly fifteen hours of real crawler traffic, none of it ours.
| First door | Before (23 h) | After (14.5 h) |
|---|---|---|
| AI crawler requests | 910 | 583 |
| Stopped at the edge | 610 | 12 |
| Share stopped at the edge | 67% | 2% |
Source: Cloudflare analytics API, cittago.com zone. The windows differ in length, so the honest comparison is the percentage, not the absolute count.
The first door is solved, and the numbers say so: 67% down to 2%. The twelve remaining blocks are the crawlers we keep out deliberately — the ones that take and give nothing back.
What is worth taking from that number is not its size but its speed. One line in one rule, and by the next morning you know whether it worked. Very little else in this field answers that fast.
What these numbers do not say
| What we measured | What we cannot claim |
|---|---|
| One site, two short windows | This is not a study. It is one case, ours, with the data on the table. |
| Refused requests | We do not know how many AI citations were lost. Nobody publishes that number. |
| Windows of 23 h and 14.5 h | Crawler traffic arrives in waves. One aggressive hour moves the percentages. |
| The geographic rule | We are not saying it was wrong. It still blocks unwanted human traffic, and it stays on. |
How to check your own site in twenty minutes
- Open the security analytics of whatever sits in front of your site and filter by crawler name: GPTBot, ChatGPT-User, OAI-SearchBot, PerplexityBot, ClaudeBot, Google-Extended, Applebot.
- Read the rules you wrote, not the general settings. Country and continent blocks are the prime suspects, especially old ones.
- Split the two doors. If a request was stopped before it reached the server, you fix it in your own panel. If it reached the server and was refused there, it is fixed on the server — yourself if you run it, otherwise through your hosting provider.
- Count bytes, not status codes. A 200 carrying 6,000 bytes where the page is 50,000 is a block wearing a success badge.
- Build the test page. A random code at the top, another at the bottom, a keyword in the middle, three numbers to add. Ask an assistant to report them exactly, then check your log for what it actually received. Five minutes to build, and it is the one piece of evidence that cannot argue with you.
One warning from experience: if you test from your own server, or from any address on an allow list, you will get a 200 no matter which bot name you send. The result looks lovely and means nothing. The only measurement that counts is real crawler traffic, read from your logs.
Where you stand, in three thresholds
If nothing sits in front of your site and there is no security layer on the server, you probably do not have this. Check once anyway; "probably" is not a measurement.
If something does sit in front and you have never opened the AI crawler analytics, you are where we were on 4 August. That does not mean you are blocked. It means you do not know, and twenty minutes settles it.
If you checked and found blocks, fix them in order: the front door first, where you have control, then the server. Re-measure after each step, or you will never know which one did the work.
On our own site the first door is fully solved and verified in numbers. On the second we have made good progress, and over the coming days it lands exactly where we want it. Ask us again in three to six months 😉
Questions
Questions nobody has actually asked us
We are one day out from the measurement, so no, nobody has asked anything yet. These are the questions a reader would have, plus the ones that come up anyway in real conversations with clients about AI visibility.
Should I even want AI crawlers on my site?
Separate the two kinds. Bots that read in order to answer a question someone is asking right now are traffic you almost certainly want — that is a customer at the door. Bots that harvest text for training are a business decision, and there are legitimate reasons to keep them out.
What is never a good answer is deciding based on the name a visitor gives you. Names are typed; verification is not.
If my robots.txt allows them, isn't that enough?
No, and this is the exact trap we fell into. robots.txt is a voluntary instruction for bots that choose to honour it. A firewall is a door. You can have "come in" painted on the wall and the door locked two metres further along.
Ours named GPTBot, ChatGPT-User, ClaudeBot and PerplexityBot explicitly, and refused 79% of their requests.
How do I tell without technical access?
Quickest signal: give an assistant the link to one of your pages and ask it to quote a specific sentence from it. If you get generalities about your company but never a verbatim line, it probably did not open the page.
That is an indication, not proof. Proof lives in the logs, and for that you need someone with access to the security panel.
Worth separating from a different problem: a page that is never indexed cannot be cited either, for reasons that have nothing to do with firewalls. We looked at that in what 150,000 pages show about AI content and indexing.
Why did Googlebot get through when the others didn't?
Geography, not favouritism. Googlebot crawls heavily from data centres in the United States and Europe, and our rule blocked other continents. A large share of AI infrastructure egresses through Singapore, which the rule caught — 630 of the blocks we measured came from there.
Same rule, opposite outcomes, for a reason that has nothing to do with content.
Does blocking AI crawlers hurt my Google rankings?
Not directly, and our own control group shows that: Googlebot was refused once in 79 requests while AI crawlers were refused 79% of the time. They are governed separately.
The risk lies elsewhere. Some crawlers serve more than one purpose, and a rule aimed at training can catch a search function too. That is why the diagnosis has to be done per crawler rather than per policy.
What does it cost to fix?
The first door is a rule change in your own firewall panel: an hour if you know what you are looking for, and nothing in subscription terms — we are on Cloudflare's free plan.
The second door is on the server. If you run it yourself it is again a matter of rules; if you do not, your hosting provider handles it.
How do I avoid breaking something else?
Save the current configuration first so you can roll back. Then widen an existing rule rather than stacking a new one on top — rules evaluate in order, and a new one in the wrong position can cancel another.
And leave the protections that guard human traffic alone. You are not opening everything; you are carving a narrow exception for verified bots.
Why doesn't the AI crawler dashboard show any of this?
Because custom rules are evaluated before it, and what they block never reaches its analytics. Cloudflare documents this, but it is easy to miss.
The practical consequence is unpleasant: the panel can report "allow" for every crawler while none of them reaches your site.
What does a block with status 200 look like?
The server says everything is fine and hands over a verification screen of a few thousand bytes instead of a page of a few tens of thousands. Ours: 6,064 bytes against 54,801, status 200.
You only catch it by comparing response size against the real page size. The status code tells you nothing here.
Is a permanent test page worth keeping?
Yes, and it is the cheapest instrument in this whole story. A random code at the top, another at the bottom, three numbers whose sum appears nowhere. If a model returns all the values, it read the page; if not, you have a concrete thing to investigate.
Ours stays public at cittago.com/ai-access-test/. Yours takes five minutes to build.
Last updated: 6 August 2026. All figures come from the Cloudflare GraphQL analytics API for the cittago.com zone, over windows ending 5 August at 17:21 UTC and 6 August at 08:35 UTC; the raw data is archived, because the free plan retains it for 24 hours. The rule change went live on 5 August at 17:33 UTC and its effect is measured over the window that follows it. We update this page when the situation changes.
