Google uses the Search foundation
Google states that AI features rely on existing Search systems. There is no special AI schema requirement; index eligibility, snippets and normal Search controls remain relevant.
OpenAI separates search from training
OAI-SearchBot supports search discovery, while GPTBot is a separate control associated with potential model training. Publishers should decide on each purpose independently.
Anthropic and Perplexity publish distinct controls
Anthropic and Perplexity document crawler identities and retrieval behaviour. Robots rules should be paired with live CDN or WAF checks because a policy file alone cannot prove successful access.
Practical checklist
- Document the purpose of each crawler rule
- Separate search retrieval from training preferences
- Test live responses with the published user agents
- Review official documentation after platform changes