Scraping vs API for AI visibility: why sampling wins

There is a real argument for capturing the consumer interface, and it isn't marketing spin. The consumer app and the developer API are different products. ChatGPT's web interface has its own system prompt, memory, personalisation, model routing and — most decisive for brand visibility — its own retrieval pipeline. Call the same model over the API and you get none of that. The same gap exists at Perplexity, whose Sonar API models behave differently from the consumer product. And for Google AI Overviews and AI Mode there is no public API at all, so capturing the rendered answer is the only option that exists. But “we measure exactly what users see” is an overreach. To scrape at scale you need proxies and either logged-out sessions or a farm of accounts. What you capture is a memoryless synthetic user on a datacenter IP — which is also an artificial condition, just a different one. It is higher fidelity than a bare API call. It is not ground truth. The quieter cost is reliability. Automated scraping of consumer chat interfaces generally runs against their terms of service. The practical consequence isn't a courtroom — it's bot detection, rate limits and silent data gaps whenever an interface ships a redesign. API access is contractual and stable. For an agency putting a number in front of a client every month, an unexplained gap is a serious problem.

Here is the fact the whole category rests on and rarely states plainly: large-language-model answers are non-deterministic, and retrieval drifts. Ask the same question twice and you can get two different answers; ask it next week and the sources have moved. Any single measurement — scraped or API — is one sample from a distribution. That makes data quality a question of sampling design, along four dimensions: 1. Runs per prompt. How many times is each prompt asked, per engine, per scan? A brand seen in one of two runs is not “50% visible” in any meaningful sense — that's a coin flip. The uncertainty of a proportion shrinks only as the sample grows (Brown, Cai & DasGupta, 2001). 2. Cadence. How often does the tool re-measure — and can you run it daily when a launch or a news cycle is moving the answer? 3. Locale coverage. AI answers are market- and language-specific. A brand's visibility in “best CRM for agencies” asked from Germany in German is a different number from the same prompt asked from the US in English. 4. Reported uncertainty. Does the number arrive with a sample size and a margin of error, or is it a bare percentage that hides how shaky it is? There is also a cost asymmetry that quietly decides the outcome. Browser-based capture costs roughly an order of magnitude more per query than an API call. So scraping-first tools tend to be sample-starved: the fidelity gain is paid for in frequency and prompt coverage. A tool that scrapes each prompt once a week is, for the purpose of a trustworthy trend, worse than one that samples the same prompt many times a week — whatever the capture method. For a metric like share of voice, sample count usually beats per-sample fidelity.

Two things make a naive AI-visibility percentage misleading, and both are fixed with honest statistics rather than more scraping. Extremes and small samples. The textbook standard error, the square root of p(1−p)/n, is known to behave badly near 0% and 100% and at small n — it can even imply an impossible visibility below 0% or above 100%, and at 0% it reports a margin of zero: false certainty that a brand is never mentioned. The Wilson score interval fixes this: it stays inside 0–100% and behaves well when the sample is thin, which is exactly where per-engine slices live. Correlated runs. Runs are not independent coin flips. They cluster hard by engine — a web-grounded model may name you often while an ungrounded one barely mentions anyone — and by day. Pooling thousands of correlated runs as if they were independent makes the margin look far tighter than it is. Accounting for that clustering (a design effect, in the language of survey sampling) widens the interval to its honest width. The practical upshot: two brands can share the same headline percentage and carry very different confidence — and a good tool shows you which is which. This is the difference between a number that looks reassuring and a number you can defend in a client meeting.

Our position follows directly from the argument above. Hybrid capture, on purpose. Where a consumer surface has no API — Google AI Overviews, Google AI Mode, Microsoft Copilot — we read the real rendered answer a searcher sees. Everywhere else we call the providers' own web-grounded APIs. API-first isn't a cost-cutting compromise; it's the low-per-query architecture that lets us sample more, and more often, which is where accuracy actually comes from. Every rate ships with its evidence. You never see a bare percentage. Each figure in CrunchJunkie carries its sample size (“n = … runs”) and a margin of error that is boundary-safe and widens when your engines disagree or the sample is thin — so a shaky number looks shaky, in the app, in reports and in shared links alike. Pooled correctly. Rates are computed as mentions over runs across the whole population, then divided once — never by averaging rates, which quietly over-weights small, lucky slices. We don't publish our exact run counts or the internals of our interval maths, but the principles are the whole point — and you should hold every vendor, us included, to them.

The honest way to compare AI-visibility tools isn't to argue about scraping. It's to ask four questions and make every vendor answer them — Peec AI, us, or anyone else: 1. How many times do you run each prompt, per engine, per scan? One run is a coin flip; a trend needs repeats. 2. How often can I scan, and is daily available on my plan? 3. Do you measure per market and per language? 4. Do you show a sample size and a margin of error — and does that margin widen when the sample is thin or the engines disagree? CrunchJunkie answers yes to all four, on every plan, across nine AI engines, with the margin of error printed next to every number. That's the ground we're happy to be compared on: not who captures the flashier surface, but whose number a client can still trust next month. If a tool shows you a confident-looking percentage with no sample size and no ±, that confidence is borrowed, not earned.

Scraping isn't bad and API sampling isn't cheating — each measures a slightly different, slightly artificial surface. The thing that decides whether your month-over-month change is signal or noise is sampling design: enough runs, often enough, in the right locale, with the uncertainty shown honestly. Ask for those numbers. The best tool is the one that can answer them.