Freeze the panel before comparing

A comparison needs compatible question versions, provider/model, method, country, language, request settings, extraction rules, business objective, and score formula. Equal date lengths alone are insufficient. Adding easier questions or changing a model can change the rate without changing your website. Preserve a stable panel and report any excluded or incompatible observations.

Repeated answers are clustered evidence

Several responses to the same question help describe variability, but they are correlated evidence about that question. They do not create several independent topics. Question-balanced aggregation keeps repeats inside their cluster. A model update, search-index change, or backend behaviour change can also affect a supposedly identical request, so repeated sampling cannot guarantee stationarity.

Frequency, first position, and top three

An optional repeated-answer experiment freezes a specific question-version and provider/request cell. Frequency counts an entity at most once per valid eligible answer. First-position and top-three measures require a defensible explicit ranking and have their own assessable population. A repeated textual mention is not an extra vote. Failed collection or failed extraction is not a negative recommendation.

Adaptive experiments need sequential uncertainty

When a sampling experiment stops after inspecting results, a fixed-sample interval should not be presented as if the sample size had been fixed in advance. The reviewed sampling implementation uses a time-uniform Beta(1,1) mixture confidence sequence for eligible Bernoulli outcomes, with error allocation across the predeclared entities, cells, and three metrics. Its validity depends on the within-cell assumptions; correlated outputs and changing search context can weaken those assumptions.

Example
Within one frozen experiment cell
n = valid eligible answers
k = answers positively recommending the entity
frequency = k / n

Beta-mixture process:
M_n(p) = B(k + 1, n - k + 1) / [p^k × (1 - p)^(n - k)]
Retain p where M_n(p) < 1 / alpha

Do not pool different models or question versions into one binomial cell.

Budgets and stopping

The sampling workflow has minimum evidence, maximum requests, and budget limits. Cells are interleaved rather than letting one provider consume the entire budget first. A stopping target is not a promise to reach a narrow interval within the request cap. Cancellation stops further dispatch where possible; a request already sent to a provider may still be billable. Optional experiments require their intended approval and are distinct from routine dashboard reads.

Pairwise preference is a separate experiment

A supported preference experiment compares a defined candidate set and scenario, including position permutations where applicable. Token log-probabilities, when actually supplied by the supported backend, are different from a model writing 'I am 90% confident'. Relative preference modelling is conditional on the candidates, scenario, model, and method. It is not a prediction that 60% of shoppers will buy a product and cannot be compared across arbitrary candidate groups.

What to report

State the panel, period, requested and valid observations, distinct questions, providers, missing assessments, method/version, and whether a value is descriptive, weighted, inferred, or experimental. Use percentage points for differences between rates. Report 'not comparable' or 'not enough data' when the required evidence is absent. A flat graph, synthetic trend, or zero-filled gap would be less informative than an honest missing value.

AEO and purchase-visibility scores: formulas and publication rulesCommerce metrics: answers, recommendation units, and sharesWeekly account synchronisation and background jobs