How ShopFlare finds, processes, and connects data

There are several data pipelines, not one universal scan#

ShopFlare stores different kinds of evidence for different jobs. Your website is a source of merchant facts. A monitored AI answer is an observation of a provider response to a particular question. Search Console reports search activity for an authorised property. An advisor event records an interaction with a published flow. Joining these records can reveal an opportunity, but does not turn one kind of evidence into another.

The common processing pattern#

Each pipeline preserves its own source and observation context. Normalisation makes evidence comparable; it should not erase the original answer or page needed to inspect a finding.

1. Select an authorised source and scope

   A workspace, connected domain, catalogue, question portfolio, or connector defines what may be processed. Availability also depends on the active package and configured provider capabilities.

2. Collect and validate

   Workers fetch allowed public pages, request configured AI answers, or read an authorised connector. Failed requests, challenge pages, incomplete extraction, and missing permissions remain explicit states.

3. Normalise and retain provenance

   Text and entities are normalised while preserving source URLs, timestamps, question versions, and relevant provider settings. Repeated evidence is deduplicated according to the consuming module's unit of analysis.

4. Index or assess

   Merchant text becomes searchable chunks. Measurement answers become evidence-backed classifications. Catalogue identities are resolved separately from whether a product was recommended.

5. Aggregate and investigate

   A card groups eligible observations using a defined denominator. Open its supporting answers, pages, products, or event breakdown before deciding what to change.

Three clocks to read on every result#

Observed time is when the source event or measurement happened. Imported or processed time is when ShopFlare received or analysed it. Displayed time is when you opened the page. A freshly loaded dashboard can contain an older catalogue snapshot. Reprocessing a saved answer can improve its classification without creating a new observation of the external model. Use source dates and measurement windows when comparing results.

An entity is not an outcome#

A product, brand, retailer, domain, category, and question are separate identities. A brand mention is not automatically a recommendation. A manufacturer recommendation is not automatically a recommendation to buy from your shop. A source link is not automatically a retailer endorsement. A confirmed catalogue identity means the observed item has been mapped to an active product record; it does not establish stock or fulfilment.

What missing values mean#

Unknown, failed, pending, excluded, and zero carry different meanings. An assessed answer that does not mention your brand may contribute a negative observation. A failed provider request cannot establish absence. An unavailable citation list cannot establish that no source was cited. A dashboard must therefore expose the eligible sample and any excluded or unresolved records alongside the rate.

Synthetic example
20 requested answers
16 completed and assessable
 4 failed or unresolved
 5 positive recommendations

Observed rate = 5 / 16 = 31.25%
Not 5 / 20, unless the metric explicitly uses requests as its denominator.

Stored evidence and generated explanations#

Counts, links, matching states, and many classifications are derived from stored records. Optional AI interpretation can summarise or prioritise that evidence, but its prose is not a new source. Keep the distinction between a measured event, an inferred topic, a generated recommendation, and a forecast. The existence of an action suggestion does not mean an edit has been made in the shop.

A reproducible investigation#

Record the metric name, workspace, selected filters, source period, provider/model, sample size, question version, and formula version where displayed. Open the supporting evidence, verify the identity and current storefront, and document any merchant change. Compare again only when the new period and collection panel are compatible. All numerical examples in these technical guides are synthetic unless explicitly identified otherwise.