AI crawlers & crawlability

Before an AI assistant can mention a brand, its crawler has to be able to read the site. CrunchJunkie gives you two complementary tools under AI Visibility → a client → the "Agent analytics" group: • Crawlability — checks a site's robots.txt to show which AI crawlers (GPTBot, ClaudeBot, Google-Extended, PerplexityBot and ~20 more) are allowed, partially blocked, or fully blocked. This is the fastest way to catch the common own-goal of accidentally blocking the very bots that feed AI answers. • Crawl insights — shows which AI bots have actually visited the site, how often, and which pages they hit, by reading your server or CDN access logs. Crawlability tells you what's allowed; Crawl insights tells you what's really happening.

Crawlability needs nothing more than the site's address. 1. Make sure the client has a website saved: Clients → open the client → set its website URL (e.g. example.com). This prefills the analyzer. 2. Open AI Visibility → the client → Crawlability. 3. The domain is prefilled (or type any domain). Click Analyze. CrunchJunkie fetches the domain's robots.txt and shows a table of known AI bots with a status — Allowed, Partial, or Blocked — plus, for each, why it's classified that way and how to change it. Every check is saved to a history timeline, so you can re-run it later and see exactly what changed since the last check. No setup, no tags — it reads the public robots.txt directly.

Crawl insights needs your server/CDN access logs, because that's the only place AI-bot visits are recorded. 1. Open AI Visibility → the client → Crawl insights. On first visit you'll see a unique ingest URL (…/api/crawl-logs/<key>) and an upload button. The key belongs to the project: if you delete a project whose drain delivered logs in the last 14 days, CrunchJunkie asks you to move the key to another project of the client first (the drain keeps working unchanged) or to confirm; a drain still posting to a deleted project's URL is answered with 410 Gone and the date, never an endless 401. 2. Get the data in, one of two ways: • One-off: download an access-log export from your server or CDN (Nginx/Apache combined format, or a Cloudflare/Vercel log export as .log/.csv) and upload the file on the page. • Continuous: point a log drain at the ingest URL so new hits stream in automatically — a Vercel log drain, an Nginx/Apache forwarder, a Cloudflare Worker (the free/Pro path; ready-to-paste snippet on the Crawl insights page), or Cloudflare Logpush (Enterprise only). One trap with a Vercel log drain: its sampling rules are a whitelist. Vercel runs them top to bottom, and a request that matches no rule is dropped. A drain whose only rule is, say, "/api/crawl-logs at 0 %" (to keep the ingest route from logging itself) therefore forwards nothing at all, and Vercel still shows it as enabled, because no delivery ever fails. If you add a rule like that, put a catch-all below it: blank path, production, 100 %. With no rules at all the drain forwards everything. 3. CrunchJunkie parses the lines, keeps only the AI-bot hits (everything else is ignored), and builds the dashboard: total visits, active bots, breakdowns by platform / bot / purpose (training, search, user-query), top crawled paths, HTTP status codes, and a visits-over-time trend you can view over 7/14/30/90 days. The top-pages table shows what real crawlers read: scanner probes (/.env-style paths under any name) and impostor requests (spoofed IPs, robots.txt-only names, IPs rotating through several operators' crawler names) are kept out and reported as one count, and a name a scanner wore is marked as such on its provider card and in the by-bot list. Under Status codes, 5xx answers are split into those a real-or-uncheckable crawler got on a content page — an outage that mattered — and those that only met scanner probes. Only AI-bot user-agents are stored — ordinary human/visitor traffic in the logs is discarded, so there's no visitor PII to manage. One card, "Asked for Markdown", depends on what your log source carries. Since spring 2026 the fetchers behind AI answers send Accept: text/markdown on ordinary page URLs, and the card shows per crawler how many fetches asked for Markdown out of the fetches whose Accept header was recorded (the sample size is always shown). The Cloudflare Worker snippet forwards the header; a Vercel log drain, Cloudflare Logpush and combined access logs do not, so with those sources the card says the header is unknown rather than showing 0 %. A custom forwarder can include the header as an "accept" field on each visit.

Crawl data isn't only for your own diagnostics — it goes straight into a client report. Add a Table widget, choose the AI Visibility source, and set its dimension to "Crawler visits". You get one row per AI crawler that actually requested pages from the site within the report's date range — GPTBot, PerplexityBot, ClaudeBot and the rest — across five columns: Bot, Platform, Type (training, search or user-query), Visits, and Share, which is that bot's percentage of all bot visits in the period. It's worth telling a client what makes this table different from everything else on the page. Every other AI-visibility figure is a sampled estimate: we prompt the models many times and count what comes back, which is the honest way to measure something non-deterministic, but it remains a sample with a margin of error. This one isn't. Each visit is a line in a server log recording that a named bot fetched a URL at a moment in time. It's a hard fact, and it's the strongest evidence you can put in front of a client that AI systems are reading their site. There's one prerequisite to get right. The table only has data if crawl-log ingestion is set up for that project (see "Set up Crawl insights" above). Without it the widget is simply empty — and empty means "not connected yet", not "no AI crawler is interested in this site". Wire up the ingest before the table goes anywhere near a client report, or it tells exactly the wrong story.

The "By site section" card groups the same page fetches as the top-pages table by the part of the site they hit, so you can see whether crawlers spend their time on /blog, /products or /docs, and where they get refused. Scanner probes, impostor requests, API or tool calls and robots.txt, sitemap and asset fetches are left out and counted under the card. A path finds its section in this order. Your own section rules come first. Then the homepage, and the first part of the path when at least one fetched page sits below it: /blog/a and /blog/b make the section /blog, and /blog itself joins it. A language folder such as /de/ or /en-gb/ is kept as a prefix, so /de/blog and /blog are two sections. A page at the top level with nothing fetched below it (/about, /pricing) goes into "Top-level pages". The twelve busiest sections are shown and the rest are added up into "Other sections", which says how many there were; a section from your own rules is always shown. Each section shows its requests and their share of all page fetches, the requests per crawler operator (OpenAI, Anthropic, Google and the others), how many came from an operator's published IP ranges, the number of distinct pages, and the share answered 2xx, refused by a firewall or CDN (403/406), or failed (404, 410, other 4xx, 5xx). Those shares are taken over the fetches whose log line carried a status, and the card gives that number. All figures are counts or shares divided once over the whole window, never averages. Your own sections: open "Edit sections" on the card and type one rule a line, a path prefix and optionally a name after an equals sign, for example /de/blog = Blog (DE). The longest matching prefix wins, so /de/blog takes its pages before /de. Up to 20 rules; they apply to the page, the "Crawl sections" report table and the MCP connector at once, for every period, because they only change how the stored fetches are grouped. A section crawled often is not a section AI answers cite. Crawlers fetch pages to index them or to answer one question; answers are usually built from an index or a cache. Read this card for access and for problems such as a firewall refusing one section, and the AI Visibility pages for citations.

If Crawlability shows an AI bot you want is Blocked, edit the site's robots.txt to allow it (the "How to change it" note on each row tells you the exact directive), deploy, then click Analyze again to confirm the change — the history timeline will record the improvement. Partial means only some paths are disallowed: rules for /api, /admin or a login area are normal and need no change; check the listed paths and change them only if they cover pages you want answers to use. If Crawl insights shows a bot returning lots of 404s or never hitting your key pages, that's a signal to fix internal links, sitemaps, or server errors so the crawler can actually index the content you want surfaced in AI answers.