--- title: "CrunchJunkie crawlers: user agents, robots.txt and contact" url: "/bot" canonical_url: "https://crunchjunkie.io/bot" markdown_url: "https://crunchjunkie.io/bot.md" language: "en" type: "page" summary: "Which crawlers CrunchJunkie runs, how they identify themselves, how they treat robots.txt, and how to allow or block them." site: "CrunchJunkie" llms_txt: "https://crunchjunkie.io/llms.txt" --- # CrunchJunkie crawlers: user agents, robots.txt and contact Source: https://crunchjunkie.io/bot # Our crawlers CrunchJunkie fetches web pages in two situations: when one of our customers audits a website they manage, and when someone runs a GEO audit of a site. This page lists every user agent they send, what they do and how to allow or block them. ## CrunchJunkieSiteAudit Fetches the pages of a website when one of our customers runs an audit of a site they manage, starting from its home page and following its links and sitemap. - It crawls only the website that belongs to the audit: its host with and without www. A start page that redirects to another site ends the crawl. - It reads robots.txt before any page, the site's own and its www twin's, and obeys it: the group for CrunchJunkieSiteAudit if there is one, otherwise the \* group. It checks robots.txt again before every redirect it follows, so a URL robots.txt closes is never requested. If robots.txt answers with a server error, it stops, as the robots.txt standard (RFC 9309) requires. - It sends at most one request a second to a site, every request counted: robots.txt, sitemaps, llms.txt, pages, redirects and the check below. It honours Crawl-delay up to 10 seconds, slows down when a server answers 429 or 503, pauses for 10 minutes after three server errors in a row, and never crawls one site twice at the same time. - It keeps what an audit needs (status codes, titles, links between pages), not the text of the pages. User agents ``` Mozilla/5.0 (Linux; Android 10; K) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Mobile Safari/537.36 (compatible; CrunchJunkieSiteAudit/1.0; +https://crunchjunkie.io/bot) Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36 (compatible; CrunchJunkieSiteAudit/1.0; +https://crunchjunkie.io/bot) ``` To check whether a firewall turns AI crawlers away although robots.txt allows them, it requests the home page once with the published user agent of each of these crawlers: OAI-SearchBot, ChatGPT-User, PerplexityBot, Perplexity-User, ClaudeBot, Claude-User, Googlebot, meta-webindexer. These requests are paced like all others. ``` Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36; compatible; OAI-SearchBot/1.3; +https://openai.com/searchbot Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; ChatGPT-User/1.0; +https://openai.com/bot Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot) Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Perplexity-User/1.0; +https://perplexity.ai/perplexity-user) Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; +claudebot@anthropic.com) Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Claude-User/1.0; +Claude-User@anthropic.com) Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) meta-webindexer/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/web-crawlers) ``` It sends no other user agent. ## CrunchJunkie-GEO Fetches the home page, robots.txt, llms.txt, the sitemap and up to nine further pages of a site when someone runs a GEO audit, our check of how ready a site is for AI search. It identifies itself as CrunchJunkie-GEO/1.0 and reads robots.txt as tikitaka-crawlability/1.0. To check whether a firewall refuses AI crawlers, and whether AI crawlers get the same content as people, it also requests the home page and one content page with a standard browser user agent and with the published user agents of AI crawlers. User agents ``` CrunchJunkie-GEO/1.0 tikitaka-crawlability/1.0 Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36 Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36 Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; ChatGPT-User/1.0; +https://openai.com/bot Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36; compatible; OAI-SearchBot/1.3; +https://openai.com/searchbot Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot) Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Perplexity-User/1.0; +https://perplexity.ai/perplexity-user) Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; +claudebot@anthropic.com) Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Claude-User/1.0; +Claude-User@anthropic.com) Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) meta-webindexer/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/web-crawlers) ``` ## Block or allow our crawler To keep the site-audit crawler out of your site, add this to your robots.txt: ``` User-agent: CrunchJunkieSiteAudit Disallow: / ``` To let it through a firewall or CDN, allow requests whose user agent contains CrunchJunkieSiteAudit. Our requests come from changing cloud IP addresses, so allow by user agent rather than by IP address. ## Contact A question about our crawlers, or a problem with them? Write to us: [hello@crunchjunkie.io](mailto:hello@crunchjunkie.io)