---
title: "We ran the same AI-visibility study twice. Eight of 19 agencies changed rank. Not one change beats the margin of error."
url: "/blog/ai-visibility-noise-vs-signal"
canonical_url: "https://crunchjunkie.io/blog/ai-visibility-noise-vs-signal"
markdown_url: "https://crunchjunkie.io/blog/ai-visibility-noise-vs-signal.md"
language: "en"
last_updated: "2026-09-23"
type: "blog"
summary: "Two windows, no shared days, 19 US agencies, the same engines and questions. Eight changed rank. Not one change is bigger than the margin of error."
site: "CrunchJunkie"
llms_txt: "https://crunchjunkie.io/llms.txt"
---

# We ran the same AI-visibility study twice. Eight of 19 agencies changed rank. Not one change beats the margin of error.
Source: https://crunchjunkie.io/blog/ai-visibility-noise-vs-signal
Published: 2026-09-23 · By Philipp Enders · AI visibility · 10 min read

On 8 September we published a ranking of US GEO agencies built from 198 AI answers. The obvious question about a ranking like that is whether it holds still. So we ran the identical measurement again, over a later stretch of days that shares no dates with the first, and put the two side by side. Eight of the nineteen agencies changed position. First and second swapped. And not one of those changes is bigger than the margin of error on samples of 168 and 90 answers. iPullRank was named in 37% of answers both times and still lost first place; in the second window it was two answers behind. If you report AI visibility to clients, the distance between what the chart shows and what the data supports is the whole problem.

## The short version
Two windows, no shared days: 4 to 8 September and 14 to 23 September. Same six buyer questions, same two engines, same nineteen agencies. Each agency was named in some share of 168 answers in the first window and some share of 90 in the second.

Eight agencies changed rank between them. iPullRank went from first to second having lost 0.2 percentage points. Siege Media took first place on a gain of 4.4. First Page Sage climbed from ninth to sixth. Yes Optimist slipped from thirteenth to fourteenth.

None of those movements is bigger than the margin of error. Every agency's range in the first window overlaps its range in the second — all nineteen of them. The lead change at the top is two answers: in the second window Siege Media was named in 35 of 90 answers, iPullRank in 33. The largest raw change in the table, Directive's 6.5 points, sits comfortably inside what a sample this size produces when nothing has happened at all.

## The same nineteen agencies, measured twice
Read the table as two separate measurements of the same thing, not as a before and after. Each range is the margin of error: the band the real share could plausibly sit in, given how many answers stand behind it. iPullRank was named in 62 of 168 answers, 37%; on that many answers the real share could be anywhere from 30% to 44%. Where one agency's two ranges overlap, the difference between its two windows is not established. (The method is the 95% Wilson interval; see the methodology note.)

The first window has 168 answers rather than the 198 the September ranking was built from because it leaves out 3 September, the day the list of tracked agencies was still being changed; the two windows compare like with like only from 4 September.

The second window is smaller, 90 answers against 168, so its ranges are wider. That is not a defect in the second window. It is what fewer answers buys.

The complete per-agency data — both windows, the 95% ranges and the split by engine — is available as a [CSV file](https://crunchjunkie.io/blog/data/ai-visibility-noise-vs-signal-2026-09.csv), published under [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/): use it freely, credit CrunchJunkie and link to this page as the source.

| Agency | 4–8 Sep (168 answers) | Margin of error (95%) | 14–23 Sep (90 answers) | Margin of error (95%) | Change |
| --- | ---: | ---: | ---: | ---: | ---: |
| iPullRank | 36.9% | 30.0–44.4 | 36.7% | 27.5–47.0 | −0.2 pp |
| Siege Media | 34.5% | 27.8–42.0 | 38.9% | 29.5–49.2 | +4.4 pp |
| Omniscient Digital | 30.4% | 23.9–37.7 | 30.0% | 21.5–40.1 | −0.4 pp |
| Directive | 28.0% | 21.7–35.2 | 34.4% | 25.4–44.7 | +6.5 pp |
| Obility | 24.4% | 18.5–31.4 | 21.1% | 14.0–30.6 | −3.3 pp |
| [OutreachBloom](https://outreachbloom.com/) | 19.6% | 14.3–26.3 | 17.8% | 11.2–26.9 | −1.9 pp |
| Go Fish Digital | 19.1% | 13.8–25.7 | 17.8% | 11.2–26.9 | −1.3 pp |
| NoGood | 19.1% | 13.8–25.7 | 18.9% | 12.1–28.2 | −0.2 pp |
| First Page Sage | 16.1% | 11.3–22.4 | 21.1% | 14.0–30.6 | +5.0 pp |
| Seer Interactive | 14.9% | 10.3–21.0 | 17.8% | 11.2–26.9 | +2.9 pp |
| WebFX | 14.9% | 10.3–21.0 | 14.4% | 8.6–23.2 | −0.4 pp |
| Foundation Marketing | 10.7% | 6.9–16.3 | 11.1% | 6.2–19.3 | +0.4 pp |
| Yes Optimist | 8.9% | 5.5–14.2 | 4.4% | 1.7–10.9 | −4.5 pp |
| Digital Elevator | 7.7% | 4.6–12.8 | 6.7% | 3.1–13.8 | −1.1 pp |
| Bay Leaf Digital | 4.2% | 2.0–8.3 | 3.3% | 1.1–9.3 | −0.8 pp |
| Citant.ai | 3.0% | 1.3–6.8 | 2.2% | 0.6–7.7 | −0.8 pp |
| Grow and Convert | 1.2% | 0.3–4.2 | 1.1% | 0.2–6.0 | −0.1 pp |
| NP Digital | 1.2% | 0.3–4.2 | 1.1% | 0.2–6.0 | −0.1 pp |
| Are You Visible | 0.0% | 0.0–2.2 | 0.0% | 0.0–4.1 | 0.0 pp |

Share of AI answers naming each agency, measured over two windows with no shared days. ChatGPT and Claude pooled, six buyer questions, United States.

## Eight changed rank, and rank is the thing people read
Nobody opens a client report and studies a margin of error. They look at the order, and the order is the least stable thing on the page, because turning a rate into a position throws the margin away.

Look at the top of the table. In the first window iPullRank led on 62 of 168 answers and Siege Media followed on 58. In the second, Siege Media led on 35 of 90 and iPullRank followed on 33. Four answers separated them the first time, two the second. One answer falling the other way on one day would have flipped it back.

That is not a story about Siege Media taking the lead from iPullRank. It is a two-answer margin being reported as a change in market leadership. The same applies further down: First Page Sage's three-place climb rests on 27 of 168 becoming 19 of 90, two measurements whose ranges almost completely overlap.

The agencies that genuinely separate are the ones far apart to begin with. iPullRank against Foundation Marketing is a real gap in both windows. iPullRank against Siege Media is not a gap in either.

![Visibility trend line chart from the CrunchJunkie AI Visibility overview, 3 to 23 September 2026, six agencies: OutreachBloom, iPullRank, Siege Media, Omniscient Digital, Directive and Obility. The six lines run between roughly 15% and 45% and cross each other repeatedly from day to day. Below the chart a note reads that 8 configuration changes fall inside the period, so a period-over-period comparison is not like for like.](https://crunchjunkie.io/blog/img/ai-visibility-noise-vs-signal-trend.webp)

## How big a move has to be before it means anything
The useful form of this finding is a number you can apply to your own reporting. For a brand sitting near 30% visibility, here is how many answers each window needs before a change of a given size can be told from chance. (Standard two-proportion test; the settings are in the methodology note.)

The first row is the one that matters. Detecting a five-point move takes roughly 1,377 answers in each window. At the rate this project runs — seven questions (the six used here plus one brand question), twice a day, on two engines, so 28 answers a day — that is about seven weeks per window, fourteen weeks to compare two. Nobody in this category reports on that cadence. We do not either.

The practical reading is not that you need fourteen weeks. It is that the movement in a weekly AI-visibility report is almost entirely noise, and that reporting it as performance is a claim the data cannot carry.

| To detect a move of… | …each window needs | …which is about |
| --- | ---: | ---: |
| 5 points, 30% to 35% | 1,377 answers | 7 weeks |
| 10 points, 30% to 40% | 356 answers | 13 days |
| 15 points, 30% to 45% | 163 answers | 6 days |
| 20 points, 30% to 50% | 93 answers | 3 days |

Answers needed in each window to detect a change from a 30% baseline, two-proportion test, 95% confidence, 80% power. Days assume 28 answers a day.

## The engines disagree far more than the weeks do
Across the combined 4 to 23 September window each engine produced 129 answers per agency. Splitting the same answers by engine produces differences that dwarf anything the calendar produced.

Seer Interactive was named in 5 of Claude's 129 answers and 36 of ChatGPT's. NoGood, 11 against 38. Omniscient Digital runs the other way, 50 against 28. Those gaps are far bigger than the margin of error; none of the week-to-week movements were.

So when an AI-visibility number moves and you want to know why, the engine mix is a better first suspect than the market. Adding or dropping an engine, or a provider swapping the model behind a product name, shifts a blended figure further than any realistic change in how often people are actually recommended. We have written separately about [why the engines disagree about your brand](https://crunchjunkie.io/blog/why-ai-engines-disagree-about-your-brand); this is the same effect, measured against the noise.

![Performance matrix table from CrunchJunkie showing AI visibility percentage by brand for ChatGPT and Claude, pooled over 362 answers from 26 August to 24 September 2026. Seer Interactive reads 27% on ChatGPT and 4% on Claude; NoGood 26% and 8%; Omniscient Digital 19% and 34%; iPullRank 44% and 27%; Siege Media 36% and 31%.](https://crunchjunkie.io/blog/img/ai-visibility-noise-vs-signal-engine-matrix.webp)

| Agency | Claude (of 129) | ChatGPT (of 129) | Gap |
| --- | ---: | ---: | ---: |
| Seer Interactive | 5 (3.9%) | 36 (27.9%) | 24.0 pp toward ChatGPT |
| NoGood | 11 (8.5%) | 38 (29.5%) | 21.0 pp toward ChatGPT |
| iPullRank | 35 (27.1%) | 60 (46.5%) | 19.4 pp toward ChatGPT |
| Omniscient Digital | 50 (38.8%) | 28 (21.7%) | 17.1 pp toward Claude |
| Go Fish Digital | 18 (14.0%) | 30 (23.3%) | 9.3 pp toward ChatGPT |
| Yes Optimist | 15 (11.6%) | 4 (3.1%) | 8.5 pp toward Claude |
| First Page Sage | 27 (20.9%) | 19 (14.7%) | 6.2 pp toward Claude |
| Bay Leaf Digital | 9 (7.0%) | 1 (0.8%) | 6.2 pp toward Claude |

Answers naming each agency, by engine, 4–23 September 2026. Each engine produced 129 answers per agency. The eight widest gaps.

## What we would put in a client report
Three changes, none of which needs a different tool.

Report the count next to the rate. "Named in 33 of 90 answers" invites the right question in a way that "36.7%" does not. A client who can see the denominator can work out for themselves that a two-answer swing is not a trend.

Put the margin of error on the chart. A bar with a range on it is harder to over-read than a bar without one, and when two competitors' ranges overlap the honest line is that they are level, not that one is ahead.

Stop reporting rank movement week to week. Report the rate, the count and the range, and let the order be whatever it is. If a client asks why the order changed, the answer is usually that it did not change in any sense worth acting on.

None of this makes the number less useful. It makes the claims attached to it survive being checked, which over a retainer is worth more.

## What this does not say
It does not say AI visibility cannot be measured. The engine differences above are real and large at this sample size, and so is the distance between an agency named in a third of answers and one named in none. Coarse facts come through clearly on a few hundred answers. Fine ones do not.

It does not say the September ranking was wrong. Both windows put broadly the same five agencies at the top and the same seven at the bottom. The structure held. Only the order inside it moved.

It does not generalise past this setup. Nineteen agencies, six questions, two engines, one market, twenty days. A category with more candidates, or with coverage that genuinely moves, could behave differently. The arithmetic about sample sizes is not specific to agencies, though. It applies to any brand whose visibility is measured as a share of answers, which is all of them.

### FAQ
**How much does an AI-visibility number have to move before it means something?**
At the sample sizes most AI-visibility reports are built on, a few hundred answers or fewer, roughly 13 to 20 percentage points. For a brand near 30% visibility, telling a 5-point move from chance takes about 1,377 answers in each of the two windows being compared (standard two-proportion test; settings in the methodology note). On 198 answers the smallest detectable move is about 13.5 points; on 90 answers it is about 20.

**Why did the ranking change if nothing changed?**
Because a rank is only the order of the percentages, and the order ignores the margin of error. Two agencies separated by two answers out of ninety will swap places routinely without either of them gaining or losing anything. In this study eight of nineteen agencies changed rank between two windows in which no agency changed by a distinguishable amount.

**Does scanning more often fix this?**
Only if it increases the number of answers inside the window you compare. Scanning daily and then comparing one day to the next leaves you with a smaller sample and a noisier number, not a better one. What helps is more questions, more runs per question, and comparing longer windows rather than adjacent short ones.

**Should we report AI visibility to clients weekly?**
Report the level weekly if the client wants to see it. Do not report the week-on-week change as performance, because at a weekly sample size almost none of that change is real. A monthly or quarterly comparison over a pooled window carries far more of the information and far less of the noise.

**ChatGPT and Claude give different lists. Which one is right?**
Both, for their own users. Over 129 answers each in this study, Seer Interactive was named by ChatGPT in 36 and by Claude in 5, while Omniscient Digital was named by Claude in 50 and by ChatGPT in 28. Neither is an error. A blended score hides that split, so it is worth keeping the per-engine numbers visible alongside it.

### Methodology & data
The same six buyer questions used in [our September ranking of US GEO agencies](https://crunchjunkie.io/blog/best-geo-agencies), asked of ChatGPT (gpt-4o-search-preview) and Claude (Sonnet 4.6, with search), twice per engine per day, market United States. Two windows that share no dates: 4 to 8 September 2026, giving 168 answers per agency, and 14 to 23 September 2026, giving 90. Where a day carried two scans, only one is counted, so no day is weighted twice. An agency counts as named when an engine writes its name in the answer body; a citation of its domain without a naming does not count. Counts are pooled across days, questions and engines first and divided once; no percentage is averaged with another percentage. Ranges are 95% Wilson score intervals on the pooled count and denominator. Sample-size figures use a two-proportion test at 95% confidence and 80% power. 3 September is excluded from both windows because the tracked brand list changed that day and the answer sets were of three different sizes. Gemini and Google AI Overviews are excluded throughout: both ran on 3 September only, against 10 of the 20 brands, which is far too thin to rank on. Measured with CrunchJunkie, which we build. The full per-agency table is published as [a CSV under CC BY 4.0](https://crunchjunkie.io/blog/data/ai-visibility-noise-vs-signal-2026-09.csv).

Dataset: https://crunchjunkie.io/blog/data/ai-visibility-noise-vs-signal-2026-09.csv (https://creativecommons.org/licenses/by/4.0/)

### Sources
- [Wilson, E. B. (1927). Probable Inference, the Law of Succession, and Statistical Inference. Journal of the American Statistical Association 22(158)](https://doi.org/10.1080/01621459.1927.10502953)
- [OpenAI — Bots and crawlers (OAI-SearchBot)](https://developers.openai.com/api/docs/bots)
- [Google Search Central — AI features and your website](https://developers.google.com/search/docs/appearance/ai-features)