Back to GEO LibraryDISCOVERY / EVIDENCE CHAIN

From crawl permission to AI recommendation

Crawl permission opens a door. It does not prove indexing, retrieval, citation or recommendation; each stage needs its own evidence.

Updated
20/08/2026
Publisher
NobleJackal
Version
1.0
Source review
Primary sources reviewed
01

Access is only the first gate

Check robots.txt, response status, CDN rules and the rendered page before discussing visibility. Search crawlers and training crawlers can have different purposes and controls, so their access should never be reported as one combined signal.

02

Indexing and retrieval are different events

A page may be crawlable yet absent from an index, or indexed without being selected for a particular answer. Internal links, visible text, canonical consistency and source quality help discovery, but they do not guarantee selection.

03

Citation and recommendation require observed proof

Record the exact prompt, market, language, model, date, response and cited URL. A citation shows source visibility in that sample. A recommendation adds a comparative judgement and must be evaluated separately.

Practical checklist

  • Verify crawler access and rendered content
  • Confirm canonical and index eligibility
  • Test retrieval with a fixed prompt set
  • Log citations and recommendations as separate outcomes

Official sources

  1. Google Search CentralAI features and your website
  2. OpenAIOverview of OpenAI crawlers
  3. IndexNowIndexNow protocol documentation