11.6% of AI Crawler Fetches Hit a Firewall. Here's What That Costs.

First the shape of the traffic, because it surprised us and it will probably surprise you. In two weeks, one mid-sized marketing site was fetched 3,695 times by AI crawlers from 12 providers — OpenAI alone accounts for 38.5% of it (1,421 fetches), with Anthropic (14.2%), Meta (12.9%) and Apple (12.7%) behind. The single most active bot was ChatGPT-User at 762 visits: not a training crawler, but the fetch ChatGPT makes live, on behalf of a user, while it is composing an answer. That last distinction runs through everything else in this article. Of all AI fetches in the window, 27.4% were user requests — a model reading your page right now because someone asked it something — and 72.0% were search-and-training traffic building tomorrow's indexes and models. Two very different reasons to be crawled, and as we'll see, two very different costs when the crawl gets blocked.

430 of those 3,695 fetches came back with HTTP 403 or 406. That combination — robots.txt allows the bot, the server refuses the request — is the signature of a firewall or CDN bot rule, and it deserves its own name: allowed but not reachable. It doesn't show up in any robots checker, because robots.txt is fine. It doesn't show up in your analytics, because the request never rendered a page. It only shows up in the server logs, in the status-code column nobody reads. The blocks were nothing like evenly distributed. ByteDance had 95.3% of its 337 fetches refused. Amazon lost 65.8%, You.com 57.1%. Meanwhile OpenAI was blocked on just 1.3% of fetches, Anthropic on 1.9%, Perplexity on 2.5%. That pattern is typical of a CDN's default bot list: the household names are whitelisted, everyone else hits the wall. Nobody at the site chose this per provider — a managed rule set chose it, silently, and the logs are the only place the choice is visible.

Here is where it stops being alarming and starts being useful. Of the 430 blocked fetches, 401 hit training or bulk crawlers — GPTBot-style traffic that builds future models and indexes. Many sites block that deliberately, and it's a defensible choice: you are trading long-term model familiarity for control over your content, and reasonable people land on both sides. The other 29 hit live-answer crawlers — the ChatGPT-User class of fetch, where a model is assembling an answer for a real person and wants to read your page right now. Those are the blocks that cost you. When that fetch fails, the engine either answers from what it already believes about you or cites someone whose page it could reach. You will never see the failure; the person asking never knows your page existed. Twenty-nine fetches in two weeks is a small leak on this site, precisely because the big live-answer crawlers were whitelisted. Flip the firewall's defaults — or install a stricter rule set — and that number becomes the whole 27.4%. If you take one action from this article: separate the two categories before you decide anything. Blocking training bots is a policy decision. Blocking live-answer crawlers is almost always an accident.

The per-provider split repays a close look, because the crawl behaviour maps to what each company sells. OpenAI's 1,421 fetches split 762 user requests against 659 search-and-training — better than half of its interest in this site was live answering. Perplexity's 277 fetches included 64 user requests. Those two providers are reading pages to cite them. Meta fetched 478 times: zero user requests, all training. Apple fetched 471 times: also zero user requests. Apple is the quiet one worth knowing about — it crawls busily for models that today cite almost nothing, so its traffic is nearly pure training draw. And Google barely appears at all: 24 fetches in fourteen days, because AI Overviews and AI Mode answer from the search index Google already has. A site can obsess over GPTBot and forget that its Google AI presence was decided years ago by ordinary indexing (we looked at those two surfaces separately in tracking visibility in AI Overviews and AI Mode).

One more thing the logs taught us: a user-agent string is a claim, not an identity. Anything can call itself GPTBot. The honest way to count AI crawler traffic is to check each fetch against the IP ranges the provider publishes — OpenAI, Anthropic and Perplexity all maintain official lists. In our window, 82.0% of AI-labelled fetches verified against a published range. OpenAI's traffic verified at 86.5%, Perplexity's at 66.1% — and the fetches claiming to be Google verified at 0%, which almost certainly means they weren't Google at all. This matters in both directions. If you rate-limit or block by user agent alone, impostors get treated as the real crawler and the real crawler inherits the impostors' reputation. And if your analytics counts "AI traffic" by user agent alone, a chunk of it is scrapers wearing a costume. Verification is not paranoia; on this evidence roughly one fetch in five needs it.

You need two things: what your server actually returned to AI crawlers recently, and whether the live-answer crawlers can reach you right now. For the first, pull two weeks of access logs and filter for the AI user agents, then look only at the status codes. Anything systematic in the 403/406 range against a bot your robots.txt allows is a firewall or CDN rule — find it in your WAF's bot-management settings, where AI crawlers are increasingly blocked by default categories rather than by anything you chose. We walked through the full log-reading exercise in The Supply Side of AI Visibility, and CrunchJunkie's Crawl Insights does it continuously from forwarded logs, including the verification step and the user-versus-training split. For the second, you don't need logs at all: our free GEO audit runs a live reachability probe of the citation crawlers against your site and tells you which ones get in, in about a minute, no signup. If a live-answer crawler is being turned away today, that is the single highest-leverage fix in AI visibility — everything else in this field assumes the engines can read you at all.