Website discovery, crawling, and knowledge indexing
Understand URL discovery, rendered-page collection, deterministic cleanup, chunking, embeddings, and the conditions that keep a source out of answers.
On this page
The workspace corpusHow URLs are discoveredRendered content and access checksCleanup preserves useful factsDuplicate pages and product variantsChunking and embeddingsAtomic replacement and reuseWhy a page may not answer a questionThe workspace corpus
Website knowledge is organised around an authorised workspace source rather than treating every widget as an unrelated website. Eligible experiences can reuse that corpus. Saving a widget does not imply a new full-site crawl. Connecting or refreshing a source is a separate operation with its own status, scope, limits, and completion time.
How URLs are discovered
Discovery starts from the configured website and selected language or locale. Bounded mapping and sitemap discovery identify candidate pages within the allowed scope. Limits on page count, depth, request size, and worker execution prevent unbounded traversal. A discovered URL is only a candidate: it must still be fetched, validated, stored, and indexed. The number of sitemap entries is therefore not the number of searchable pages.
Rendered content and access checks
The crawler uses rendered page content so that useful text produced by a supported storefront can be collected. The pipeline checks network access and the returned document. An HTTP 200 response can still be a bot challenge, consent wall, or empty shell. Such a response must not become product knowledge merely because the request succeeded. Permanent missing pages and temporary upstream failures are handled differently; transient failures have bounded retries. Access restrictions are not bypassed.
| Situation | Interpretation | Useful merchant check |
|---|---|---|
| Discovered, not fetched | URL exists in the queue, not yet in the knowledge index | Check job progress and selected scope |
| Challenge or access-denied document | The expected source was not obtained | Allow legitimate crawler access using the installation guidance |
| 404 or 410 | Source is unavailable at that address | Correct the URL or remove the outdated source |
| Saved, processing incomplete | Raw content exists but may not be searchable yet | Inspect indexing status rather than repeatedly refreshing |
| Ready and eligible | The source can participate in retrieval | Test a question whose answer is present on that page |
Cleanup preserves useful facts
The deterministic cleanup stage normalises text, removes scripts and presentation markup, reduces navigation and consent noise, and recognises repeated blocks. Repetition alone is not a reason to remove a fact: shipping conditions, return rules, safety information, and other recurring policy material can be important. Original source material and search-oriented text serve different purposes. The reviewed cleanup version is deterministic-v2-corpus; a version change can require reindexing without refetching the website.
Duplicate pages and product variants
Canonical duplicate handling reduces repeated copies of essentially the same document. A near-duplicate product variant is not automatically the same product: size, colour, revision, connector, capacity, or regional variant may change the answer. Canonical and exclusion metadata are checked during retrieval as well as indexing, so an old vector must not make an excluded page usable again.
Chunking and embeddings
Cleaned content is split into pieces with a reviewed target budget of 2,800 characters including title and source-URL prefixes. Tables carry useful headers across pieces. A page producing more than 500 chunks fails explicitly instead of silently truncating its remaining facts. These are implementation limits, not a promise that every page reaches the limit. The embedding service turns each piece into a numeric vector; the reviewed index validates 768 finite dimensions. A vector is a semantic lookup representation, not a compressed authority on the product. The original text is still needed to substantiate an answer. A document that exceeds processing limits should have an explicit error rather than silently losing an arbitrary tail.
Atomic replacement and reuse
Index updates coordinate access to the corpus and affected pages, then replace eligible chunks transactionally. Unchanged content can reuse existing work. A failed or pending replacement must not be advertised as ready. Keeping a previous source record is useful for diagnostics, but does not mean an incomplete new version should be served as current. An embedding outage can leave lexical retrieval available for eligible saved text; it does not authorise cross-workspace searching.
Why a page may not answer a question
Check the whole chain: was the correct locale discovered, was the actual page fetched, did it contain the needed fact, did processing finish, is it canonical and eligible, and can retrieval find the relevant passage? A successful crawl does not guarantee every attribute is present. A ready index does not guarantee every passage fits into one model request. Fix the narrowest missing step instead of starting repeated full-site imports.
Still need help?
Tell us what you expected and what happened.