40% off every plan. See pricing

Skip to content
Veritas
Try Free
Glossary · Letting AI crawlers in

AI crawler

Also called AI bot, LLM crawler, GPTBot, ClaudeBot, PerplexityBot

Definition

An AI crawler is a bot run by an AI company that fetches web pages, either to collect training data for models or to find and read sources for answers.

Updated 15 Sept 20264 sources, each checked3 min read

The idea

What it is

An AI crawler is any automated agent an AI company sends to your site. They are not one kind of bot. Each major vendor now runs separate agents for separate jobs, and each announces itself with its own user-agent token, so your robots.txt can treat them differently.

The useful split is by purpose. Training crawlers collect pages that may end up inside a future model. Search crawlers build the index an assistant draws on when it answers with sources. User-triggered fetchers visit a page because a person asked the assistant about it right now. Blocking one of these tells you nothing about the other two.

The evidence

Why it matters

Whether an assistant can cite you depends on whether its search crawler or fetcher can read you. Plenty of sites blocked "AI bots" wholesale when training crawlers first appeared and, in doing so, also shut out the agents that decide which pages get linked in answers. The vendors have since separated those agents precisely so that sites can say no to training and yes to being cited.

The vendors that sell AI visibility show how settled this has become. In our scan of their own robots.txt files, none blocks any AI agent from the whole site, and most never name an AI agent at all, leaving them to the default rule.

The steps

How to do it

  1. 1

    List the agents that matter to you and what each one is for. Training: GPTBot (OpenAI), ClaudeBot (Anthropic). Search and citation: OAI-SearchBot (OpenAI), Claude-SearchBot (Anthropic), PerplexityBot (Perplexity). User-triggered fetches: ChatGPT-User, Claude-User and Perplexity-User.

  2. 2

    Decide your policy per purpose, not per company. The common position for a business that wants to be cited is to allow the search agents and fetchers, and to allow or disallow the training agents as a separate decision.

  3. 3

    Write one robots.txt group per agent you treat differently, naming the exact token, and leave everything else to your wildcard group.

  4. 4

    Check the result by fetching your live robots.txt and reading which group each agent actually falls into. A typo in a token silently sends that agent to the wildcard rule.

  5. 5

    Remember that robots.txt is a request, not a lock. User-triggered fetchers may not follow it, so anything that must stay private needs authentication, not a disallow line.

Run this check on your own site

Free, no sign-up.

AI crawler checker
What goes wrong

Common mistakes

  • Blocking every AI user agent to stop training, which also removes you from the search indexes that produce cited answers.
  • Blocking the search agent of one vendor and assuming the others follow, when each vendor reads only the group that names its own token.
  • Treating a disallow rule as privacy. It asks compliant crawlers to stay away and does nothing about the rest.
In practice

An example

Allow citation, opt out of training

A robots.txt that lets assistants cite a site while declining model training has a group for GPTBot with "Disallow: /", a group for ClaudeBot with "Disallow: /", and no groups for OAI-SearchBot, Claude-SearchBot or PerplexityBot, so those fall through to a wildcard group that allows the public site. Each vendor documents that its search agent is separate from its training agent, which is what makes the split possible.

OpenAI, Overview of OpenAI crawlers, 15 Sept 2026
Nearby terms

Not to be confused with

Search engine crawler
Googlebot and Bingbot build a ranked index of links. AI crawlers feed a model or an answer, and one vendor may run several with different purposes, so a single rule rarely fits them all.
robots.txt
An AI crawler is the visitor. robots.txt is the file that tells compliant visitors, by user-agent token, which paths they may fetch.
Original data

Do AI search vendors let AI crawlers in?

VendorNames AI crawlersBlocks any AI crawler site-widePublishes llms.txt
Veritas (ours)getveritas.ioYesNoYes
Semrushsemrush.comNo, wildcard ruleNoYes
Frasefrase.ioYesNoYes
Profoundtryprofound.comNo, wildcard ruleNoNo
Jasperjasper.aiYesNoYes
Surfersurferseo.comNo, wildcard ruleNoNo
Writesonicwritesonic.comNo, wildcard ruleNoYes
Otterly.AIotterly.aiUnknownUnknownUnknown
Scrunchscrunch.comNo, wildcard ruleNoYes
AthenaHQathenahq.aiNo, wildcard ruleNoYes
GEOforgegetgeoforge.comNo, wildcard ruleNoNo
Passionfruitgetpassionfruit.comYesNoYes
Ahrefsahrefs.comNo, wildcard ruleNoNo
SE Rankingseranking.comYesNoNo
Peec AIpeec.aiNo, wildcard ruleNoNo

How we checked. One fetch of each domain’s /robots.txt and /llms.txt on 15 September 2026. For each AI agent we read the group that names it, or the wildcard group if none does, and recorded whether the whole site is disallowed. An llms.txt counts when the path returns a plain-text file rather than an HTML page.

  • A single fetch on one day with one user agent. Vendors change these files without notice.
  • robots.txt states a policy. It does not show what a site actually enforces at the server.
  • Where a server refused our request, the row is marked unknown rather than guessed.
Checked

Sources

  1. 1OpenAI, Overview of OpenAI crawlers read 15 Sept 2026
  2. 2Anthropic, Does Anthropic crawl data from the web, and how can site owners block the crawler? read 15 Sept 2026
  3. 3Perplexity, Perplexity crawlers read 15 Sept 2026
  4. 4Veritas crawler policy scan of 15 AI search vendors read 15 Sept 2026
Where this lives in VeritasCheck which AI crawlers your site allowsAI crawler audit: the bots, explained
Related termsrobots.txtrobots.txt is a plain-text file at the root of a site that tells crawlers, by user-agent name, which URLs they may and may not fetch.llms.txtllms.txt is a proposed markdown file at a site's root that gives language models a short summary of the site and links to the pages most worth reading.AI citationAn AI citation is a source link an answer engine attaches to its generated answer, crediting a page as the origin of the information it used.AI visibilityAI visibility is how often, and how favourably, a brand appears in the answers AI engines generate for the questions its buyers ask.Generative Engine Optimization (GEO)Generative Engine Optimization (GEO) is the practice of shaping content so AI answer engines quote it and name it as a source in their answers.

Content that is built to be cited.

Every draft grounded in your own documents, with each knowledge-based claim cited to its source. 7 days free on one domain.

Start your 7-day free trial See pricing