The AI Crawler Audit: Sixteen Bots and What Each Does With Your Pages

Two jobs, and a third kind of visit

"Block the AI bots" is not one decision. The crawlers AI companies operate do different jobs, and their operators document them separately.

• Search agents index pages so an answer engine can surface and cite them.
• Training agents collect content that may be used to train models.
• User-initiated fetchers load a page because a person asked a question that needs it.

The mistake we see most is a robots.txt that blocks everything with an AI company's name on it. It opts the site out of training, which may be intended, and out of citation, which usually is not. The sixteen agents below are the ones our AI crawler checker reads, plus a few more worth knowing, each described from its operator's own documentation where one exists.

The search agents: whether you can be cited

The consequences of blocking these are stated plainly. OpenAI's documentation says "Sites that are opted out of OAI-SearchBot will not be shown in ChatGPT search answers." Anthropic's says blocking Claude-SearchBot prevents it indexing your content for search. Google's guidance on AI features says a page must be "indexed and eligible to be shown in Google Search with a snippet" to appear in AI Overviews or AI Mode, so blocking Googlebot removes you from both along with ordinary search.

Meta documents a similar agent, Meta-WebIndexer, which it says "navigates the web to improve Meta AI search result quality for users."

The user-initiated fetchers: when someone asks

These visit a single page because a person's question needs it. They are where robots.txt behaves least like people expect.

• ChatGPT-User is "used for certain user actions in ChatGPT." OpenAI says that "Because these actions are initiated by a user, robots.txt rules may not apply," and that it is "not used to determine whether content may appear in Search."
• Perplexity-User supports "user actions within Perplexity." Perplexity says it "generally ignores robots.txt rules."
• Claude-User fetches pages when people ask Claude questions. Anthropic says its agents honour robots.txt, and that blocking Claude-User prevents it "retrieving your content in response to a user query."
• Meta-ExternalFetcher fetches individual links at a user's request, and Meta notes it may bypass robots.txt for that reason.

If a page must not be read by these agents, robots.txt is not a reliable control for all of them. Authentication is.

The training agents: whether your content trains models

• GPTBot (OpenAI) is "used to crawl content that may be used in training our generative AI foundation models." OpenAI says "Disallowing GPTBot indicates a site's content should not be used in training," and that "Each setting is independent of the others."
• ClaudeBot (Anthropic) collects web content "that could potentially contribute to their training." Blocking it "signals that the site's future materials should be excluded from our AI model training datasets." Some robots.txt files still name anthropic-ai, an older token; Anthropic's current documentation lists ClaudeBot, Claude-User and Claude-SearchBot.
• Google-Extended lets publishers manage whether content may be used "for training future generations of Gemini models that power Gemini Apps and Vertex AI API for Gemini and for grounding." Google is explicit that it "does not impact a site's inclusion in Google Search nor is it used as a ranking signal in Google Search."
• Applebot-Extended lets publishers opt out of their content being used to train Apple's foundation models. Apple notes that "Applebot-Extended does not crawl webpages" and that pages disallowing it "can still be included in search results."
• CCBot is the crawler of Common Crawl, a non-profit that maintains "an open repository of web crawl data," which many models are trained on.
• Meta-ExternalAgent "crawls the web for use cases such as training foundation AI models or improving products by indexing content directly."
• Bytespider (ByteDance) and Amazonbot (Amazon) also appear in many robots.txt files. We did not find documentation for either that we could verify for this post, so we describe neither here; the checker shows what your file says to them.

One more OpenAI agent is worth knowing about so it does not surprise you in your logs: OAI-AdsBot validates pages submitted as ads on ChatGPT. OpenAI says it only visits pages submitted as ads, that robots.txt does not apply, and that its data is not used for training.

Why naming a bot can remove your rules

This is the robots.txt behaviour that catches the most people. RFC 9309, the Robots Exclusion Protocol standard published in September 2022, says a crawler looks for a group that names it, and only "If no matching group exists, crawlers MUST obey the group with a user-agent line with the '*' value."

So if your file has Disallow: /admin/ under User-agent: *, and you add a group that says User-agent: GPTBot with Allow: /, GPTBot no longer reads the wildcard group at all. Your admin rule does not apply to it. Any bot you name needs every rule it should follow repeated in its own group. Where rules conflict, the RFC says "The most specific match found MUST be used," meaning the longest matching path wins.

A robots.txt that allows citation and opts out of training

A starting point, to adapt to your own paths. The standard allows several User-agent lines to share one group. The robots.txt generator builds the same kind of file from the one you have now, with citation and training as two separate decisions.

User-agent: *
Disallow: /admin/

User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
Disallow: /admin/

User-agent: GPTBot
User-agent: ClaudeBot
User-agent: anthropic-ai
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: CCBot
User-agent: Meta-ExternalAgent
Disallow: /

Googlebot falls under the wildcard group here, so it keeps crawling for Search and its AI features. The search agents are named only to make intent explicit; notice they repeat the admin rule, for the reason in the previous section.

Two caveats. A training opt-out is a request that the operators above say they honour; it does not reach crawlers that publish nothing about their behaviour. And it is forward-looking, as Anthropic's wording about "future materials" makes clear.

The five-minute audit

1. Open your robots.txt, or paste it into the AI crawler checker, which sorts every agent into allowed, blocked and not mentioned.
2. For each search agent, confirm it can reach the pages you want cited.
3. For each training agent, confirm the setting matches a decision someone actually made.
4. For every agent named in its own group, confirm the group repeats the rules from the wildcard group that should still apply.
5. Write the decision down, so the next person who edits the file knows which lines were deliberate.

For the wider picture of what these engines do once they can reach you, see How to Get Cited in ChatGPT, Perplexity, and Google AI Overviews and How Perplexity Chooses Its Sources. The glossary entry on AI crawlers has the short version.