The Supply Side of AI Visibility: How to Read Your AI Crawl Logs

An AI-visibility score answers one question: how often does ChatGPT mention us? It is the result of a sampling process — we ask the engines many times and count — and it is genuinely useful. But it is a lagging outcome. When the number is low, the score itself can't tell you whether the model never saw your page, saw it but found nothing worth quoting, or saw a competitor's page instead. The crawl log is the supply side. It reads your server or CDN access logs, keeps only the hits from known AI crawlers — GPTBot, PerplexityBot, ClaudeBot and the rest — and shows how many times each AI system fetched you, which pages, and with which HTTP status. It is not a sample or an estimate: every line is a real request a named bot made at a real timestamp, and where a provider publishes its official crawler IP ranges each hit is verified against them — 100% verified on this project — so a spoofed user-agent can't inflate the count. That makes it the most concrete evidence you can put in front of a client that AI systems are reading their site — and the first place to look when the visibility score disappoints. The rule of thumb: the score tells you the outcome; the crawl log tells you whether the precondition for that outcome is even met. A model cannot cite a page it never successfully fetched.

Not every AI fetch means the same thing, and the split between two kinds is the single most useful number on the page. OpenAI documents three separate crawlers, and the distinction is the whole game (OpenAI, "Overview of OpenAI's crawlers"). GPTBot collects content to train future models. OAI-SearchBot builds OpenAI's search index. ChatGPT-User is different: it fires when a real person asks ChatGPT something and the assistant fetches your page live to answer them — and, per OpenAI's own documentation, includes a link to the source in its response. Each is an independent robots.txt setting; you can welcome one and refuse another. Group them and you get two buckets. Search-and-training crawlers are building a picture of your site over time — passive, slow-moving, necessary but not urgent. User-request fetches are the leading indicator that actually matters: a person asked a question, and an engine went to your page to answer it, right then. That is the fetch most likely to end in a citation. On pmax.online over 14 days, 82.4% of AI fetches (89 of 108) were search-and-training, and just 12.0% (13) were user requests. That ratio is the thing to move, and it reads two ways. If your user-request fetches climb while your visibility stays flat, you have a content problem — engines are coming to answer real questions and finding nothing to lift. If user requests sit near zero, you have a discovery problem — engines aren't reaching you at the moment of the question at all. Thirteen live retrievals in two weeks is a large part of why our own visibility sits where it does.

The next panel splits fetches by provider, and the shape is almost always lopsided. On pmax.online: OpenAI 72.2%, ByteDance 9.3%, Anthropic 7.4%, Google 5.6%, Amazon 2.8%, Apple 1.9%, Meta 0.9%. When one provider dominates and the others barely register, the instinct is to blame your content. It is usually infrastructure. Amazonbot, Applebot and Meta-ExternalAgent are exactly the crawlers a CDN's default AI rule switches off. Cloudflare shipped a one-click "Block AI Scrapers and Crawlers" toggle to every plan, free tier included, in July 2024 — and its block list names Amazonbot, Applebot, ChatGPT-User and ClaudeBot specifically (Cloudflare, "Declaring your AIndependence"). A robots.txt someone wrote two years ago does the same quietly. So a near-zero Amazon, Apple or Meta share usually means one thing: you are invisible to those surfaces regardless of how good your pages are, because a bot rule is turning them away at the door. (Here Anthropic sits at a healthy 7.4% — half of it live Claude-User fetches — which is exactly what a reachable crawler looks like, and the contrast is the tell.) This is a five-minute check that beats a month of content work. Before you conclude a provider doesn't like your content, confirm its crawler can reach you at all: check robots.txt for that specific user-agent, and check your CDN's bot rules. Low provider share is a plumbing diagnosis first and a content one second.

The status-code panel is the one most people scroll past, and on our own site it held the finding that mattered most. Of 108 AI fetches: 200 (OK) 63, 304 (Not Modified) 26, 406 (Not Acceptable) 13, 301 (redirect) 5, 400 (Bad Request) 1. Thirteen of a hundred and eight — about 12% — returned HTTP 406. A 406 means the server refused the request based on its headers, a classic signature of a WAF or CDN edge rule blocking a bot that robots.txt actually allows. This is the "allowed but not reachable" trap: our robots.txt said yes, our edge said no, and the crawler left empty-handed roughly one time in eight. No sampled visibility metric would ever have explained that shortfall. The log did, in one line. Status codes catch what sampled metrics can't: bots getting 403s or 406s from your WAF, slow responses timing out, or — the quiet killer — content that only renders after JavaScript. Most live-answer crawlers execute little or no JavaScript, so a client-rendered page is a blank page to them even when it returns a clean 200. If your AI fetches are landing but your content still isn't being quoted, that is where to look. Fix the transport before you touch the words. (We later ran this exact analysis at thirty times the scale — 3,695 fetches over fourteen days — and the pattern held: 11.6% of AI-crawler fetches hit a firewall, with the blocks concentrated on providers the CDN's default list doesn't whitelist.)

The last panel lists the pages AI crawlers actually pulled. On pmax.online the top of the list was robots.txt (10), the homepage (6), a single blog post (4), and sitemap.xml (3) — and then it thinned out fast. That distribution is a map of what engines can find. If crawlers only ever touch your homepage, your blog index and your robots file but never your comparison pages, your pricing, or your service and category pages, those pages are either poorly linked internally or aren't being surfaced as answer candidates. The pages you most want cited — the ones that actually convert — are the ones missing from this list. Cross-reference it with the questions you want to win, and the gap tells you exactly where to write next and what to link to it from. A crawler that only ever reads your front door has only ever seen your front door.

Publishing a page is not the same as a model knowing it exists. The crawl log closes that loop. It carries a heartbeat — on our project, "receiving logs: 46 lines in the last batch, this minute" — so you can confirm the feed is live, then watch what happens after you ship. Publish or materially update a page and watch whether re-fetches follow within days or drag out over weeks; a fast re-crawl means your changes are reaching the models, a slow one is a nudge to tighten sitemap hygiene and internal linking. Publish, watch the re-fetch, adjust — that is how a passive log becomes an operating rhythm. Two caveats keep the report honest, and both are worth saying out loud to a client. First, crawl volume is a proxy, not a result: a page can be fetched constantly and never cited, and a page fetched rarely can be quoted every time. Access is the precondition for a citation, not the same thing — which is exactly why a serious programme needs both halves, the crawl log for supply and the sampled visibility score for the outcome. Don't let a rising fetch count stand in for rising visibility. Second, allowing training crawlers is a strategy call, not a technical default. Blocking GPTBot keeps your content out of the next model's training set; it does not remove you from today's answers, which are served by the search and user-request crawlers. Some brands deliberately allow retrieval bots and block training ones. The crawl log is what lets you make that decision with numbers instead of a guess.

So what does a 14-day log of 108 AI fetches actually tell us about pmax.online? Mostly that the volume is low, and that the first fixes are not content fixes. A hundred and eight fetches in two weeks is a thin diet for a site that wants to be cited across seven engines. But before writing a single new page, the log points at two cheaper wins: the 12% of fetches returning 406 (a firewall rule to relax) and the near-zero Amazon, Apple and Meta shares (crawler access to confirm). Those are plumbing, and plumbing is where missing AI-crawler volume usually hides. That is the value of reading the supply side first. The visibility score tells you that you're not being cited enough. The crawl log tells you whether it's because the models can't reach you, can't read you, or genuinely have nothing to quote — and those three problems have completely different fixes. Start with the log, in CrunchJunkie's Crawl Insights, and you fix the cheap, structural problems before you spend a quarter writing content that a firewall was going to block anyway.