AI crawler controls by platform

Separate public editorial and commercial pages from account areas, intake payloads, private files, preview hosts, internal search results and administrative routes. robots.txt is not access control: a route containing confidential material belongs behind authentication even when every crawler is disallowed.

Record the rights position for books, datasets, images and client material. A crawler policy cannot grant rights the publisher does not hold, and a permissive site-wide rule should not silently expose material whose publication status is uncertain.

Write a small decision table: agent, provider, purpose, desired policy, owner, date, source URL and review date. Search access, user retrieval and model development should not inherit one another merely because they come from the same company.

If the objective is discoverability, preserve the automatic search agents needed for that objective. A publisher may still choose a different model-development policy. State the trade-off plainly and obtain the appropriate legal or rights decision where the content warrants it.

Publish exact agent groups without accidental precedence conflicts. Then inspect CDN bot rules, managed challenges, country blocks, rate limits, origin allow-lists and application middleware. An Allow line cannot override a firewall that returns 403, a challenge page or an empty 200 response.

Use bounded rate controls. A harsh blanket rule can make a platform abandon retrieval, while no protection at all can expose the origin to spoofed traffic. Where a provider publishes IP ranges, combine those ranges with the declared user-agent and refresh them from the official endpoint.

First test whether the target URL is allowed by the relevant robots group. Then send a named-agent request from outside the origin environment and retain status, final URL, headers, body hash and challenge result. Finally render representative pages to ensure the answer, evidence and links survive.

A synthetic user-agent test proves only how the stack responds to that string from the test network. It does not authenticate a platform visit. Genuine log classification requires the provider's current IP publication or its prescribed DNS verification method.

A green delivery test establishes technical eligibility at one time. Origin logs may later show a verified visit; webmaster tools may show crawl or indexing; answer observations or platform reports may show a citation. None of these stages should be backfilled from another.

Review the policy when official documentation, crawler names, IP feeds, firewall products, hosting topology or publication rights change. Save the former version and retest before reporting the new state as live.