# Do AI crawlers really get blocked? Reading the test

> We gave 45 crawler situations to Sonnet and Haiku. Bare bot names fooled both, and the smaller model also slipped on robots.txt. Each trap and its fix.

Published 2026-10-20 · https://aiskills402.com/blog/ai-crawler-access-test-misread

A site can be said to block an AI crawler only after you sent the vendor's complete user-agent string, once per bot of each company (its trainer and its search fetcher), plus a control request that resembles a normal browser. A bare name such as `GPTBot` proves nothing, because sites answer it differently from the real crawler. Models fall for this: handed such results, they declared that a site treats one company's crawler differently from another's. More traps sit around Cloudflare defaults, old HTTP clients and a token that is not a crawler at all; the models walked into some of them and stepped around others. Below are the method, each trap, and an honest account of how much two of our skills changed the scores.

## Method

The first test is 23 pasted curl transcripts, written by us as an auditor might paste them. In 15 the tempting reading is wrong; in the other 8 it is correct, which lets a harmful skill show itself. Each reply was JSON: a status for training, search and agent bots, a likely cause, and what robots.txt looked like. A script graded only the fields where one answer is defensible.

The second test is 22 robots.txt wishes in plain English, about two thirds hiding a trap. We ran each returned file through a parser that follows [RFC 9309](https://www.rfc-editor.org/rfc/rfc9309) and asked, bot by bot, whether the stated paths were reachable.

Sonnet and Haiku each took every case twice: bare, then with one of our skills in front. We then went through every miss by hand, since a red cell can be a wrongly labelled field instead of a wrong idea.

## Results

### The headline scores

Reading probes: Sonnet went from 19 to 23 of 23 and Haiku from 17 to 21. Writing robots.txt: Sonnet went from 21 to 22 of 22 and Haiku from 18 to 20. So the stronger model gains four cases on probes, one of them only a mislabelled field, and a single disputed line on the file. The smaller one gains on both.

### A bare bot name is not a probe

In our measurement across 82 sites in September 2026, a bare `ClaudeBot` got a 403 where a bare `GPTBot` got a 200, and yet with the full strings every one of those sites treated both alike. Filters match on the string they expect, and a lone word is not it. That is how a careless test ends up writing "this site shuts out Anthropic and welcomes OpenAI".

In our case of exactly this shape both models called training mixed. In its twin, where three bare names all returned 200, both called training open. The defensible answer in each is "unknown". Both fixed this when the skill came first.

A usable probe pairs the example string OpenAI documents for its training bot ([OpenAI's crawler page](https://developers.openai.com/api/docs/bots)) with a browser string as the control:

```bash
GPT='Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot'
BROWSER='Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/141.0.0.0 Safari/537.36'
curl -s -o /dev/null -w "gptbot %{http_code}\n"  -A "$GPT"     https://example.com/
curl -s -o /dev/null -w "control %{http_code}\n" -A "$BROWSER" https://example.com/
```

A refused control means the result tells you nothing about AI crawlers. Haiku, with no help, saw a refused control plus three refused OpenAI bots and wrote "blocked" for all three kinds; "unknown" was the right word.

Anthropic's help page names ClaudeBot, Claude-User and Claude-SearchBot but gives no complete strings, so take those from your own logs; the page points to a published IP list for confirming who a visitor is ([Anthropic's page](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler)).

### A refusal can be a default, not a decision

Cloudflare's post of 15 September 2026 describes separate Search, Training and Agent settings, and says zones that never touched the finer controls were migrated according to their old block switch ([Cloudflare's post](https://blog.cloudflare.com/accountable-mixed-use-ai-crawlers/)). On our own zones that had no earlier blocks, training bots began to get 403 that afternoon. A refused training bot may therefore be a choice or merely a default. If a search bot is turned away, suspect a defect.

One case had a zone untouched since August, where the agent bot was refused only on pages carrying ads. Sonnet blamed an owner rule; the expected cause was the Cloudflare setting, and Haiku answered "unknown".

### Old clients are refused by a different gate

Clients such as Python-urllib, libwww-perl and Java 1.x receive a 403 page from Browser Integrity Check, which looks at headers and unusual user agents ([Cloudflare's description](https://developers.cloudflare.com/waf/tools/browser-integrity-check/)), whereas curl, node and current Java pass. If your audit script uses one of those refused clients, the "block" belongs to the script. Both models handled our cases on this correctly even without help.

### A missing robots.txt can answer 200

On our zones, a missing file came back as a 200 with comments only, carrying the headers of the 404. Check for a `User-agent:` line before claiming the site has a policy. Both models passed these cases bare.

### Google-Extended is a token, not a crawler

Google calls it a standalone product token that never shows up as a user agent, so a request sent "as Google-Extended" measures nothing ([Google's crawler page](https://developers.google.com/search/docs/crawling-indexing/google-common-crawlers)). Both models wrote "open" for training after such a probe. Sonnet's own explanation said the probe proves nothing, so that miss is a wrong label, not missing knowledge.

### Writing the file: one group, no Host

A bot follows the most specific group that names it. Google says a group for one bot is not merged with the star group ([Google's robots.txt page](https://developers.google.com/search/docs/crawling-indexing/robots/robots_txt)), and the RFC sends a crawler to the star group only when nothing names it. So once a `GPTBot` group exists, the private paths of the star group have to be repeated in it. Google supports just four fields, which leaves `Host` ignored. In the policy that asked for a preferred host for its Yandex visitors, Sonnet wrote `Host` on its own, and Haiku wrote it both times. We scored the line as a miss. It is the softest call in the whole test, since the request named Yandex and Google merely skips the line, and it is everything Sonnet gained on robots.txt.

Haiku's other baseline misses: it blocked a page that the policy wanted dropped from Google results, it gave no comment about noindex in a second case, and it left private notes open to the user-triggered agents. The page problem is old: robots.txt cannot remove a page from Google's index, and noindex works only on a page the crawler may fetch, because a blocked one never shows it the tag ([Google on noindex](https://developers.google.com/search/docs/crawling-indexing/block-indexing)). We went through the same trap, with Search Console steps around it, in an [earlier report on indexing fixes](/blog/google-indexing-fixes-models-get-wrong), which matters if your real goal is removal and not blocking.

### What each gain really is

On probes, [crawler-access-probe](/skills/crawler-access-probe) lifted Sonnet by four cases: two bare-name readings, the default-versus-owner cause, and the Google-Extended field with the wrong label (so three gaps and one label). Haiku had the same four misses plus the refused control and a search-setting block it called "unknown". With the skill it repaired all six but lost two others: one reply was not valid JSON, and another wrote "unknown" in the cause field while its own text said a Cloudflare setting was probable.

On files, the strong model gains that one disputed `Host` line and the smaller one gains two cases. [robots-txt-policy](/skills/robots-txt-policy) fixed Haiku's wrongly blocked page and the open private notes, and it wrote the missing comment, but `Host` stayed, and in that same comment case Haiku now left open the thank-you pages the policy asked to block.

## What we did not measure

- **Single exposure.** Every model met every case just once, so we cannot estimate noise; one different answer shifts a score by one.
- **Our own cases.** We wrote all 45 (23 plus 22), so they match traps we already knew, and the vendor strings in them are the documented examples.
- **Vendors change behaviour.** Cloudflare moved defaults in September 2026 ([its post](https://blog.cloudflare.com/accountable-mixed-use-ai-crawlers/)), and OpenAI says its string version may change. The pages were read on 20 October 2026; if one disagrees with what we wrote, believe the page. Our 403 and 82-site results date from September and were not repeated.
- **One judgement call.** The `Host` case counts against a model that did what the request literally asked. Leave it out and Sonnet's gain on robots.txt is nothing.
- **Scored fields only.** The grader reads a few fields, so a good explanation beside a wrong label fails, and a thin one beside a right label passes.
- **No live network.** The models read pasted text and never sent a request; a real audit adds timing, redirects, caching and IP checks.
- **Two models only.** Nothing here speaks for other models or for long conversations.