We ran the same AI-visibility study twice. Eight of 19 agencies changed rank, and none of them moved.

Two windows, no shared days: 4 to 8 September and 14 to 23 September. Same six buyer questions, same two engines, same nineteen agencies. Each agency was named in some share of 168 answers in the first window and some share of 90 in the second. Eight agencies changed rank between them. iPullRank went from first to second having lost 0.2 percentage points. Siege Media took first place on a gain of 4.4. First Page Sage climbed from ninth to sixth. Yes Optimist slipped from thirteenth to fourteenth. None of those movements is distinguishable from sampling noise. Every agency's 95% range in the first window overlaps its range in the second — all nineteen of them. The largest raw change in the table, Directive's 6.5 points, sits comfortably inside what a sample this size produces when nothing has happened at all.

Read this as two independent estimates of the same quantity, not as a before and after. The ranges are 95% Wilson intervals: the band inside which the real rate plausibly sits, given how many answers stand behind it. Where one agency's two bands overlap, the difference between its two windows is not established. The second window is smaller, 90 answers against 168, so its bands are wider. That is not a defect in the second window. It is what fewer answers buys.

Nobody opens a client report and studies a confidence interval. They look at the order, and the order is the least stable thing on the page, because turning a rate into a position throws the margin away. Look at the top of the table. In the first window iPullRank led on 62 of 168 answers and Siege Media followed on 58. In the second, Siege Media led on 35 of 90 and iPullRank followed on 33. Four answers separated them the first time, two the second. One answer falling the other way on one day would have flipped it back. That is not a story about Siege Media taking the lead from iPullRank. It is a two-answer margin being reported as a change in market leadership. The same applies further down: First Page Sage's three-place climb rests on 27 of 168 becoming 19 of 90, two estimates whose ranges cover each other almost entirely. The agencies that genuinely separate are the ones far apart to begin with. iPullRank against Foundation Marketing is a real gap in both windows. iPullRank against Siege Media is not a gap in either.

The useful form of this finding is a number you can apply to your own reporting. For a brand sitting near 30% visibility, here is how many answers each window needs before a change of a given size can be told from chance, at the conventional 95% confidence and 80% power. The first row is the one that matters. Detecting a five-point move takes roughly 1,377 answers in each window. At the rate this project runs — seven questions, twice a day, on two engines, so 28 answers a day — that is about seven weeks per window, fourteen weeks to compare two. Nobody in this category reports on that cadence. We do not either. The practical reading is not that you need fourteen weeks. It is that the movement in a weekly AI-visibility report is almost entirely noise, and that reporting it as performance is a claim the data cannot carry.

Across the combined 4 to 23 September window each engine produced 129 answers per agency. Splitting the same answers by engine produces differences that dwarf anything the calendar produced. Seer Interactive was named in 5 of Claude's 129 answers and 36 of ChatGPT's. NoGood, 11 against 38. Omniscient Digital runs the other way, 50 against 28. Those gaps are wide enough to survive the sample size, which none of the week-to-week movements were. So when an AI-visibility number moves and you want to know why, the engine mix is a better first suspect than the market. Adding or dropping an engine, or a provider swapping the model behind a product name, shifts a blended figure further than any realistic change in how often people are actually recommended. We have written separately about why the engines disagree about your brand; this is the same effect, sized against the noise floor.

Three changes, none of which needs a different tool. Report the count next to the rate. "Named in 33 of 90 answers" invites the right question in a way that "36.7%" does not. A client who can see the denominator can work out for themselves that a two-answer swing is not a trend. Put the uncertainty on the chart. A bar with a range on it is harder to over-read than a bar without one, and when two competitors' ranges overlap the honest line is that they are level, not that one is ahead. Stop reporting rank movement week to week. Report the rate, the count and the range, and let the order be whatever it is. If a client asks why the order changed, the answer is usually that it did not change in any sense worth acting on. None of this makes the number less useful. It makes the claims attached to it survive being checked, which over a retainer is worth more.

It does not say AI visibility cannot be measured. The engine differences above are real and large at this sample size, and so is the distance between an agency named in a third of answers and one named in none. Coarse facts come through clearly on a few hundred answers. Fine ones do not. It does not say the September ranking was wrong. Both windows put broadly the same five agencies at the top and the same seven at the bottom. The structure held. Only the order inside it moved. It does not generalise past this setup. Nineteen agencies, six questions, two engines, one market, twenty days. A category with more candidates, or with coverage that genuinely moves, could behave differently. The arithmetic about sample sizes is not specific to agencies, though. It applies to any brand whose visibility is measured as a share of answers, which is all of them.