Direct answer
Crawling occurs when a bot or automated tool discovers a URL, requests it and attempts to retrieve the content served there. If a system cannot reach the correct page, it cannot use the current information on that page. Access still does not guarantee indexing, citation or recommendation.
In plain language
Suppose the correct information is written on a shop door. A visitor cannot read it if they cannot find the street, the door is locked or security stops them by mistake. Reaching the door still does not mean they will make a purchase.
Why this matters
An access barrier can make even excellent content invisible. The opposite response—removing security and allowing every bot unrestricted access—is also wrong. Crawler purposes and permissions should be managed separately.
Do not confuse
- Crawling: visiting a URL and requesting its content.
- Indexing: processing content into a searchable record.
- Retrieval: selecting a relevant record for a particular question.
- Training: a model-development process; it is not the same as crawling.
What should you do?
- Test anonymous access and HTTP status codes for important URLs.
- Review robots.txt, firewall, rate-limit and bot-verification rules separately.
- Use only current official provider documentation for bot names and purposes.
- Do not treat search access and training access as the same policy.
- Design a narrow allowance for verified bots instead of removing a security rule.
How do you audit it?
- Does the URL return status 200 with the actual content?
- What does robots.txt tell the relevant user agent?
- Does the CDN or security layer block a verified bot?
- Were the HTML, PDF and JSON surfaces tested separately?
- Was the result recorded with the date, IP or agent context and response body?
Limit
Crawler access may be a necessary condition; it is not a guarantee of indexing, retrieval, citation, training or recommendation. Bot names and provider policies can change over time.
Remember in one sentence
Reaching the door is the first step; being admitted and selected are different steps.
Sources for this record
- S03Google Search Central, *Overview of Crawling and Indexing*Provider documentation
- S04Google Search Central, *Block Search Indexing with noindex*Provider documentation
- S09OpenAI, *Overview of OpenAI Crawlers*Provider documentation
- S10Anthropic, *Does Anthropic Crawl Data from the Web?*Provider documentation