Our crawlers

CrunchJunkie fetches web pages in two situations: when one of our customers audits a website they manage, and when someone runs a GEO audit of a site. This page lists every user agent they send, what they do and how to allow or block them.

CrunchJunkieSiteAudit

Fetches the pages of a website when one of our customers runs an audit of a site they manage, starting from its home page and following its links and sitemap.

It crawls only the website that belongs to the audit: its host with and without www. A start page that redirects to another site ends the crawl.
It reads robots.txt before any page, the site's own and its www twin's, and obeys it: the group for CrunchJunkieSiteAudit if there is one, otherwise the * group. It checks robots.txt again before every redirect it follows, so a URL robots.txt closes is never requested. If robots.txt answers with a server error, it stops, as the robots.txt standard (RFC 9309) requires.
It sends at most one request a second to a site, every request counted: robots.txt, sitemaps, llms.txt, pages, redirects and the check below. It honours Crawl-delay up to 10 seconds, slows down when a server answers 429 or 503, pauses for 10 minutes after three server errors in a row, and never crawls one site twice at the same time.
It keeps what an audit needs (status codes, titles, links between pages), not the text of the pages.

User agents

Mozilla/5.0 (Linux; Android 10; K) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Mobile Safari/537.36 (compatible; CrunchJunkieSiteAudit/1.0; +https://crunchjunkie.io/bot)
Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36 (compatible; CrunchJunkieSiteAudit/1.0; +https://crunchjunkie.io/bot)

To check whether a firewall turns AI crawlers away although robots.txt allows them, it requests the home page once with the published user agent of each of these crawlers: OAI-SearchBot, ChatGPT-User, PerplexityBot, Perplexity-User, ClaudeBot, Claude-User, Googlebot, meta-webindexer. These requests are paced like all others.

Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36; compatible; OAI-SearchBot/1.3; +https://openai.com/searchbot
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; ChatGPT-User/1.0; +https://openai.com/bot
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot)
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Perplexity-User/1.0; +https://perplexity.ai/perplexity-user)
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; +claudebot@anthropic.com)
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Claude-User/1.0; +Claude-User@anthropic.com)
Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
meta-webindexer/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/web-crawlers)

It sends no other user agent.

CrunchJunkie-GEO

Fetches the home page, robots.txt, llms.txt, the sitemap and up to nine further pages of a site when someone runs a GEO audit, our check of how ready a site is for AI search. It identifies itself as CrunchJunkie-GEO/1.0 and reads robots.txt as tikitaka-crawlability/1.0. To check whether a firewall refuses AI crawlers, and whether AI crawlers get the same content as people, it also requests the home page and one content page with a standard browser user agent and with the published user agents of AI crawlers.

User agents

CrunchJunkie-GEO/1.0
tikitaka-crawlability/1.0
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; ChatGPT-User/1.0; +https://openai.com/bot
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36; compatible; OAI-SearchBot/1.3; +https://openai.com/searchbot
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot)
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Perplexity-User/1.0; +https://perplexity.ai/perplexity-user)
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; +claudebot@anthropic.com)
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Claude-User/1.0; +Claude-User@anthropic.com)
Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
meta-webindexer/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/web-crawlers)

Block or allow our crawler

To keep the site-audit crawler out of your site, add this to your robots.txt:

User-agent: CrunchJunkieSiteAudit
Disallow: /

To let it through a firewall or CDN, allow requests whose user agent contains CrunchJunkieSiteAudit. Our requests come from changing cloud IP addresses, so allow by user agent rather than by IP address.

Contact

A question about our crawlers, or a problem with them? Write to us: hello@crunchjunkie.io