AI crawlers & crawlability

Before an AI assistant can mention a brand, its crawler has to be able to read the site. CrunchJunkie gives you two complementary tools under AI Visibility → a client → the "Agent analytics" group: • Crawlability — checks a site's robots.txt to show which AI crawlers (GPTBot, ClaudeBot, Google-Extended, PerplexityBot and ~20 more) are allowed, partially blocked, or fully blocked. This is the fastest way to catch the common own-goal of accidentally blocking the very bots that feed AI answers. • Crawl insights — shows which AI bots have actually visited the site, how often, and which pages they hit, by reading your server or CDN access logs. Crawlability tells you what's allowed; Crawl insights tells you what's really happening.

Crawlability needs nothing more than the site's address. 1. Make sure the client has a website saved: Clients → open the client → set its website URL (e.g. example.com). This prefills the analyzer. 2. Open AI Visibility → the client → Crawlability. 3. The domain is prefilled (or type any domain). Click Analyze. CrunchJunkie fetches the domain's robots.txt and shows a table of known AI bots with a status — Allowed, Partial, or Blocked — plus, for each, why it's classified that way and how to change it. Every check is saved to a history timeline, so you can re-run it later and see exactly what changed since the last check. No setup, no tags — it reads the public robots.txt directly.

Crawl insights needs your server/CDN access logs, because that's the only place AI-bot visits are recorded. 1. Open AI Visibility → the client → Crawl insights. On first visit you'll see a unique ingest URL (…/api/crawl-logs/<key>) and an upload button. 2. Get the data in, one of two ways: • One-off: download an access-log export from your server or CDN (Nginx/Apache combined format, or a Cloudflare/Vercel log export as .log/.csv) and upload the file on the page. • Continuous: point a log drain (Cloudflare Logpush, a Vercel log drain, or an Nginx/Apache forwarder) at the ingest URL so new hits stream in automatically. 3. CrunchJunkie parses the lines, keeps only the AI-bot hits (everything else is ignored), and builds the dashboard: total visits, active bots, breakdowns by platform / bot / purpose (training, search, user-query), top crawled paths, HTTP status codes, and a visits-over-time trend you can view over 7/14/30/90 days. Only AI-bot user-agents are stored — ordinary human/visitor traffic in the logs is discarded, so there's no visitor PII to manage.

Crawl data isn't only for your own diagnostics — it goes straight into a client report. Add a Table widget, choose the AI Visibility source, and set its dimension to "Crawler visits". You get one row per AI crawler that actually requested pages from the site within the report's date range — GPTBot, PerplexityBot, ClaudeBot and the rest — across five columns: Bot, Platform, Type (training, search or user-query), Visits, and Share, which is that bot's percentage of all bot visits in the period. It's worth telling a client what makes this table different from everything else on the page. Every other AI-visibility figure is a sampled estimate: we prompt the models many times and count what comes back, which is the honest way to measure something non-deterministic, but it remains a sample with a margin of error. This one isn't. Each visit is a line in a server log recording that a named bot fetched a URL at a moment in time. It's a hard fact, and it's the strongest evidence you can put in front of a client that AI systems are reading their site. There's one prerequisite to get right. The table only has data if crawl-log ingestion is set up for that project (see "Set up Crawl insights" above). Without it the widget is simply empty — and empty means "not connected yet", not "no AI crawler is interested in this site". Wire up the ingest before the table goes anywhere near a client report, or it tells exactly the wrong story.

If Crawlability shows an AI bot you want is Blocked or Partial, edit the site's robots.txt to allow it (the "How to change it" note on each row tells you the exact directive), deploy, then click Analyze again to confirm the change — the history timeline will record the improvement. If Crawl insights shows a bot returning lots of 404s or never hitting your key pages, that's a signal to fix internal links, sitemaps, or server errors so the crawler can actually index the content you want surfaced in AI answers.