Why AI Visibility Tools Give You Different Scores

Run the same brand through two AI visibility trackers on the same afternoon and you will very likely get two different numbers. Not slightly different — sometimes twenty points apart. The instinct is to assume one tool is broken, or that somebody is inflating a dashboard. Usually neither is true.

The reason is structural. Language models do not rank pages into a list you can look up twice. They sample a sequence of tokens from a probability distribution, one token at a time. Two tools that ask slightly different questions, a different number of times, of a different mix of models, and then average the results differently, will land on different numbers even when both are measuring honestly. That is arithmetic, not malfunction.

What follows is why the disagreement happens, how large it can legitimately get, and a seven-question audit you can run on any vendor — including this one — before you treat their number as a fact.

TL;DR:

• Language models sample answers instead of ranking pages, so two honest tools can legitimately sit 20 points apart on the same brand.
• Precision is sample-size arithmetic: ±10 points at 95% confidence needs about 100 independent runs; 5 runs gives ±43.8.
• The trustworthy tool is the one whose number you can reconstruct — exact prompts, run counts, model IDs, grounding config, formula, and window.
• Track deltas on one system of record; never blend scores from different methodologies into one chart.
• The seven-question vendor audit below works on any tool, including this one.

Sampling, not ranking: where the variance actually comes from

Classic search measurement had a fixed object to point at. Position 4 was position 4; two rank trackers checking the same keyword from the same location should agree, and when they didn't, one of them had a bug.

Generative answers have no such object. A model produces text by repeatedly choosing a next token from a probability distribution, and the decoding strategy decides how that choice is made — greedy decoding takes the single highest-probability token, while temperature, top-k and top-p sampling deliberately introduce spread. Change the temperature from 0.2 to 0.7 and you change the distribution the answer is drawn from, which can change which brands get named. The same guide warns explicitly against comparing models decoded with different settings without disclosing them; the same warning applies to comparing visibility scores.

You might expect that pinning everything down — temperature 0, same prompt, same model — would collapse the variance to zero. It doesn't. OpenAI's own documentation on reproducible outputs with the seed parameter describes the seed parameter as making "a best effort to sample deterministically," and adds the crucial caveat: "There is a small chance that responses differ even when request parameters and system_fingerprint match, due to the inherent non-determinism of our models." Mostly identical is not identical.

The clearest explanation of why comes from Thinking Machines Lab's write-up, Defeating Nondeterminism in LLM Inference. Their finding: "the primary reason nearly all LLM inference endpoints are nondeterministic is that the load (and thus batch-size) nondeterministically varies." Inference kernels are not batch-invariant, so the floating-point reduction order shifts with how many other requests happen to be in flight alongside yours. In their experiment on Qwen3-235B at temperature 0, 1,000 identical requests produced 80 unique completions, with all of them diverging at token 103. Swapping in batch-invariant kernels made all 1,000 identical — which proves the nondeterminism is an engineering artifact rather than a law of nature, but also that it is present in the production endpoints every visibility tool queries today.

Academic work says the same thing from the measurement side. Song et al.'s NAACL 2025 paper, The Good, The Bad, and The Greedy, opens by noting that "Current evaluations of large language models (LLMs) often overlook non-determinism, typically focusing on a single output per example." A companion line of work, Non-Determinism of "Deterministic" LLM Settings, measured repeated runs across eight tasks and reported "accuracy variations up to 15% across naturally occurring runs" and "a gap of best possible performance to worst possible performance up to 70%." More recently, Quantifying non deterministic drift in large language models (January 2026) ran repeated trials on gpt-4o-mini and llama3.1-8b at temperatures of 0.0 and 0.7 and found that "nondeterminism persists even at temperature 0.0, with distinct variability patterns by model size, deployment, and prompt type."

Now stack a business metric on top of that. Mention rate is a proportion: out of N sampled answers, how many named your brand? If the underlying answer is a coin flip that lands "mentioned" 55% of the time, and one tool flips it five times while another flips it seventy, they will report different mention rates. Both are correct measurements of the same coin. Neither is the coin.

Five places two honest tools legitimately diverge

Every design decision below is defensible. Each one also moves the number.

1. The prompt set

Two tools rarely track the same prompts, and even when the topics match, the wording doesn't. "Best GEO tools" and "what is the best generative engine optimization software for agencies" retrieve different pages and produce different brand lists. Prompt sets also differ in composition: a set weighted toward branded queries ("is [brand] any good?") will produce a far higher mention rate than one weighted toward unbranded category queries, because your brand is already in the question.

Then there is normalization. Does a mention of "Acme" count when the model writes "Acme Corp" or "acme.com"? Does a mention inside a numbered list of eleven alternatives count the same as a mention in the opening sentence? Alias lists and prominence rules are quiet but decisive.

2. Run count

This is the biggest single lever and the least advertised. A tool that runs each prompt once per engine per day is measuring one draw from a distribution. A tool that runs it several times and averages is measuring the distribution. The first number will bounce; the second will not. Neither vendor is lying, but only one of them can tell you whether a five-point drop is a real change.

3. Engine mix and model version

If Tool A covers ChatGPT and Gemini and Tool B covers ChatGPT, Gemini, Claude and Grok, the composite scores are measuring different populations. Grok is the sharpest example, because its retrieval draws on X posts as well as the general web — a source mix no other engine shares.

Model version matters just as much as engine identity. "ChatGPT" is a product name covering several model families with different retrieval stacks and different default behavior. A tool benchmarking a fast, cheap model and a tool benchmarking the flagship the consumer app actually serves are answering different questions. Geoptimizer splits this deliberately: standard scans run the four engines through official APIs, while a Premium scan re-runs the same prompts on the flagship models the consumer apps ship with, precisely so the fidelity difference is visible rather than hidden.

4. Grounding and retrieval configuration

Whether web search was enabled — and which web search — changes both the answer and the citations attached to it.

Google's Gemini grounding documentation explains that when the google_search tool is enabled, "the model handles the entire workflow of searching, processing, and citing information automatically," returning groundingMetadata containing webSearchQueries, groundingChunks (the source URIs and titles) and groundingSupports linking response text back to those sources. Anthropic's web search API works differently: "Every web-sourced response includes citations to source materials," and organizations can configure allow lists that ensure "results only come from approved sources" or block lists that keep certain domains out. xAI's web search tool docs expose citations through the API response as well.

Three retrieval stacks, three citation formats, three different sets of domains that can end up credited. A tool with grounding on will report a different citation rate from one without it — and two tools that both enable it can still differ if one restricts domains.

There is a further wrinkle: API answers and consumer-app answers are not the same thing. App surfaces layer on personalization, memory, and UI-level product logic that APIs don't expose. Any vendor claiming their score is exactly what a user sees in the app is claiming more than the plumbing supports. Geoptimizer publishes this limitation directly on its methodology page — API answers match app answers closely, but not exactly — which is the honest position rather than a flattering one.

5. Aggregation, weighting and windowing

Even with identical raw observations, the headline number depends on how you fold them together.

Otterly.ai's KPI documentation defines a set of distinct metrics: Brand Coverage ("the percentage of tracked prompts where your brand appears"), Share of Voice against competitors, Brand Sentiment on a −100 to +100 scale, Domain Citations, and a composite Brand Visibility Index that sorts brands into Leaders, Niche or Low-Conversion segments. It's a clean, well-documented framework that works well if you want to read each KPI separately.

Geoptimizer takes a different route and folds four components into one number, with the weights published: EngineScore = 100 × (0.35·MentionRate + 0.25·CitationRate + 0.20·Prominence + 0.20·Sentiment), and VisibilityScore = mean(EngineScore) across the four engines. Prominence uses a square-root decay so an earlier mention is worth more than a late one, and the formula is versioned, so a weighting change ships as a new version rather than silently rewriting history. The reasoning behind each weight is laid out in the evidence behind our formula.

Here's the point: a Brand Coverage percentage and a weighted 0–100 composite are not the same quantity. Both approaches are legitimate; they simply cannot be diffed against each other.

Windowing is the final divider. A snapshot reports what one scan saw. A rolling window reports a smoothed estimate over repeated samples. Geoptimizer's methodology page states it plainly: "AI answers are nondeterministic — the same prompt can name different brands on different runs… The headline score is computed over a rolling window (default 7 days of accumulated samples — weekly scheduled refreshes on paid plans plus any on-demand scans) with a confidence band that narrows as samples accumulate." A snapshot and a seven-day mean of that same snapshot will disagree, by design, and the snapshot will disagree more.

How many runs do you actually need?

This is where the disagreement stops being philosophical and becomes calculable.

Mention rate and citation rate are proportions, so the standard sample-size arithmetic for proportions applies. Penn State's statistics course notes give the formula: n = (z² × p̂(1−p̂)) / ε², where z ≈ 1.96 for 95% confidence and ε is the margin of error you are willing to accept. Using the worst case p̂ = 0.5 and running the arithmetic backwards from a given N gives you the precision that N buys:

Independent runs (N)	Margin of error, 95% confidence
5	±43.8 percentage points
10	±31.0
30	±17.9
100	±9.8
384	±5.0

Read the top row again. If a tool samples a prompt five times and reports "your brand is mentioned 60% of the time," the honest reading of that number is somewhere between 16% and 100%. It is not wrong. It is just very wide, and the dashboard almost never says so.

This is the law of large numbers doing its ordinary work: "the average of the results obtained from a large number of independent random samples converges to the true value, if it exists." More samples, tighter estimate. There is no shortcut, and no amount of interface polish substitutes for N.

Two honest caveats. First, most tools do not hit N = 384 per prompt, because every run costs money and latency — vendors reasonably trade precision against price. Second, the table assumes independent draws of a single prompt, and real tools accumulate N across prompts, engines and days instead. Twenty-five prompts × four engines × seven daily samples is 700 observations, which stabilizes a portfolio-level mention rate nicely while any individual prompt still rests on seven draws. That is why prompt-level rows deserve more skepticism than the headline, and why a confidence band is worth more than a second decimal place.

The industry is converging on this. A July 2026 review of the GEO platform market notes that one tool, CiteLens, "applies a 95% Wilson confidence interval to every visibility score, a response to a core technical problem: AI answers vary from run to run, so a single-percentage-point swing can be statistical noise rather than a real trend." That is the right direction of travel.

For a concise, 2026-era explanation of what Generative Engine Optimization actually means and how the discipline maps to platforms, read a 2026 definition of Generative Engine Optimization.

The seven-question audit

Send these to any vendor, including the one you already pay. The answers, not the score, tell you what the number is worth.

1. Can I export the exact prompt text you ran? You want plain text, plus any alias or normalization rules. Small wording changes change answers. If the prompt set is a black box, you cannot tell whether a score reflects your market or the vendor's phrasing habits.

If you need a practical method to build that prompt set, see How to Choose the 25 Prompts You Track for AI Visibility.

2. How many independent runs per prompt per engine, and can I see the raw answers? Ask for N and the underlying counts behind each rate, not just the rate. A good answer is a specific integer plus access to the stored answer text. A vague answer ("we sample regularly") means you cannot compute a margin of error.

3. Which engines, and which model IDs? "ChatGPT" is a brand, not a measurement target. Ask for the exact model identifiers logged with each run, and whether the tool queries official APIs or scrapes consumer app surfaces.

4. Was web grounding enabled, and how is it configured? Ask to see citation metadata for a handful of sample answers. If domains are allow-listed or blocked, that shapes the citation rate directly.

For a short, practical guide to granting or blocking crawler access with llms.txt, see llms.txt and AI Crawler Access: The 20-Minute Setup.

5. What sampling parameters were used? Temperature, top-p, top-k, seed, and whether system_fingerprint is recorded. A vendor that has never thought about this has not thought about variance.

6. What is the exact aggregation formula, and is it versioned? How are components weighted, how are engines combined (mean, weighted mean, median), and does a formula change ship as a new version with a changelog? Without versioning, a "score improvement" might just be a weighting tweak.

7. Is the headline a snapshot or a window — and where is the confidence band? On-demand scans should be labeled snapshots. Headline numbers should carry an interval. A single run presented as the number, with no interval, is the one pattern worth pushing back on.

If a vendor answers all seven, you can reconcile their number with anyone else's. If they answer none, the number isn't wrong — it's just unaudited, which means you cannot tell wrong from right.

Reading two disagreeing scores without panicking

A worked illustration. Suppose Tool A reports 68 and Tool B reports 54 for the same brand on the same day.

Tool A runs 25 prompts once each across two engines, no grounding, and reports the share of prompts where the brand appeared anywhere in the answer. Tool B runs the same 25 topics — reworded — daily across four engines with web search on, and reports a weighted composite of mention rate, citation rate, prominence and sentiment averaged over seven days.

These numbers are not in conflict, because they are not the same measurement:

• Tool A's denominator is prompts; Tool B's is prompt-runs across a week.
• Tool A counts presence; Tool B discounts a mention buried at the bottom of a list and penalizes negative framing.
• Tool B includes two additional engines where the brand is weaker, which drags the mean down.
• Tool A's single-run design means its 68 carries a wide interval that nobody printed.

Reconciling them means decomposing, not averaging. If Tool B publishes components, you can check whether the gap lives in citation rate (a technical and authority problem) or prominence (a content-shape problem). If neither publishes components, you have two opaque numbers and no way to learn anything from their difference. That decomposability is the practical argument for open formulas: figures you can check are figures you can argue with.

For a practical walkthrough of how to compute and interpret competitive share‑of‑answers across engines, see Competitive AI Visibility Analysis: Share of Answers.

What to do with the number once you trust it

Track deltas, not absolutes. An absolute score is only meaningful inside one tool's coordinate system. Movement over time, measured the same way each time, is the signal.

Pick one tool as the system of record. Use others for cross-checking qualitative output — which sources get cited, how your category is described — but do not blend scores from different methodologies into one chart. That chart is meaningless.

Demand an interval before you act on a drop. If the confidence band on last week's 61 and this week's 56 overlap, nothing has been demonstrated. Wait for more samples or run a deeper scan.

Treat prompt-level rows as directional. Portfolio-level rates stabilize far faster than individual prompts.

Keep expectations calibrated. A critical survey of generative engine optimization published on 15 July 2026 reviewed 45 studies from November 2023 to July 2026 and found "low source overlap, substantial run-to-run variability, and persistent fidelity gaps" in commercial audits. Its recommendation is a reproducible protocol "based on repeated measurements, paraphrases, controls, human validation" — a fair description of what a well-designed visibility tracker should already be doing.

The measurement gap is widely felt. Semrush's 2026 AI Visibility Index, published 26 June 2026 from 126 million US AI search prompts across ChatGPT, Gemini, Google AI Mode and AI Overviews, reports that 45% of marketing leaders cannot accurately measure their brand visibility within AI-generated answers, and only 9% have tools that track all the relevant metrics across platforms. Under those conditions, the differentiator between tools is not who reports the highest number. It is who shows their work.

FAQ

Does a different score mean one of the tools is broken? Almost never. Given nondeterministic inference, different prompt sets, different run counts, different engine mixes and different aggregation formulas, identical scores from two tools would be more surprising than different ones. Check methodology before assuming a defect.

Which number should I actually trust? The one whose methodology you can reconstruct. If a vendor publishes prompt sets, run counts, model IDs, grounding configuration, the aggregation formula and a version changelog, you can verify the number and explain movements in it. If they publish none of that, the score may still be accurate — you just have no way to check.

How many runs per prompt is enough? For a stable single-prompt mention rate, more than most tools do: ±10 percentage points at 95% confidence needs about 100 independent runs. In practice, tools accumulate confidence across prompts, engines and days instead. The practical test is whether the vendor shows you a confidence band at all.

Should I run two visibility tools in parallel? It's useful for qualitative cross-checking — comparing which sources each engine cites, or how your category gets described. It is not useful for scoring, because the two headline numbers measure different quantities and cannot be reconciled without both formulas.

Why do scores move when I changed nothing? Three ordinary causes: sampling noise from nondeterministic inference, a shift in what the retrieval layer surfaced that day, or a model version update on the provider's side. A rolling window absorbs the first; per-run model ID logging lets you diagnose the third.

The number is only as good as the method behind it

Two tools, two scores, and no villain. AI answers are drawn, not ranked, so any single measurement is a sample with an interval around it — and the interval is usually wider than the dashboard admits. What separates a number you can budget against from a number you can only screenshot is whether the method is inspectable: the prompts, the run count, the engines and model IDs, the grounding setup, the formula, and the window.

Run the seven questions on whatever you're using now. If you want a baseline to compare against, Geoptimizer's free AI Visibility Check shows live, per-engine results across ChatGPT, Gemini, Claude and Grok in about 30 seconds with no signup, and the full scoring formula, weights, windowing and version history sit on the public methodology page so you can audit the number before you believe it. That's the standard worth holding every tool to — this one included.