Blocked pages, robots rules, and crawl access
Diagnose website access limitations without bypassing security controls or assuming that blocked content has been indexed.
Recognise a genuine access limitation
A bot challenge, login wall, robots restriction, rate limit, or server error can prevent usable content from being collected. A challenge page is not product information. Repeated retries may make rate limiting worse.
Work with your website administrator
For a site you own, ask the administrator to review the affected URLs and relevant security logs. If approved, arrange narrowly scoped crawler access with support. Keep protection on login, checkout, and private routes; do not broadly disable your firewall.
Use an approved alternative
Where an authorised upload or other supported source is available, use it for the missing information. Tell support which pages are blocked and when the crawl ran. Successful indexing cannot be guaranteed for third-party websites or deliberately restricted content.
Dashboard labels may currently be in German. Available controls and modules depend on your package and release. If a control described here is missing, check your workspace or contact support.
Still need a hand?
Tell us what you’re trying to do. Please leave out passwords and personal customer data.