Two different stages

Crawling retrieves accessible website content. Indexing prepares that content so it can be found for a question. A page can be discovered without being successfully fetched, and fetched content may still be waiting to become usable knowledge.

Interpret counts carefully

Pages discovered, processed, excluded, and failed are different measures. Duplicates, redirects, language versions, or excluded URLs can change totals. A large page count is not a guarantee of high-quality answers, and a percentage alone does not prove complete coverage.

When a status looks wrong

Check the connected website, latest crawl time, and page details before retrying. If the dashboard says no pages are indexed despite a completed crawl, send support the workspace, time, and visible status. Do not delete sources merely to make the counter reset.

Technical details and methodology

For the collection steps, eligibility rules, calculation units, and limits behind this workflow, use the implementation-reviewed technical guides below.

Website discovery, crawling, and knowledge indexingHow chatbot retrieval finds and grounds an answer