Recognise a genuine access limitation

A bot challenge, login wall, robots restriction, rate limit, or server error can prevent usable content from being collected. A challenge page is not product information. Repeated retries may make rate limiting worse.

Work with your website administrator

For a site you own, ask the administrator to review the affected URLs and relevant security logs. If approved, arrange narrowly scoped crawler access with support. Keep protection on login, checkout, and private routes; do not broadly disable your firewall.

Use an approved alternative

Where an authorised upload or other supported source is available, use it for the missing information. Tell support which pages are blocked and when the crawl ran. Successful indexing cannot be guaranteed for third-party websites or deliberately restricted content.