Who Blocks AI Crawlers? We Checked the robots.txt of 2,000 Top Domains

What we found

On 29 October 2026 we requested the robots.txt file of 2,000 domains: the first 1,000 in the world ranking that answered our request, and the first 1,000 on India's .in domain that did. For each one we checked 17 crawlers, from OpenAI's GPTBot to Googlebot, and recorded whether the file shuts each one out of the whole site.

Four results stand out.

1. About one big domain in seven blocks GPTBot. 15.0% of the global top 1,000 do. Of the Tranco top 100 domains we could reach, 13 of 71 do.
2. Training crawlers are blocked far more often than AI search crawlers. 21.4% of global domains block at least one training crawler; 12.2% block at least one AI search crawler.
3. Splitting training from search depends on the company. Most sites that block Google's AI token keep Googlebot. For OpenAI it is half, and for Anthropic fewer than half.
4. .in domains block training crawlers about half as often and AI search crawlers about a third as often, and almost none block AI search on purpose.

Training crawlers: blocked by about one domain in seven

Training crawlers collect content that may be used to train AI models. They are the most blocked agents in both lists.

Google-Extended is not only about training: Google says it also controls whether Gemini uses a site for grounding answers.

The spread is narrow because sites block these as a set. A global domain that blocks any training crawler blocks a median of 6.5 of the 9, and an .in domain blocks all 9.

AI search crawlers: blocked less often

Search crawlers decide whether a site can be found and cited inside an AI answer. OpenAI is explicit about the difference: its crawler documentation says sites opted out of OAI-SearchBot "will not be shown in ChatGPT search answers" (though they "can still appear as navigational links"), while disallowing GPTBot "indicates a site's content should not be used in training".

68 global domains block all three AI search crawlers. 43 of them block every training crawler too, and 21 block Googlebot as well, so for many of these domains it is one blanket rule rather than a decision about AI search.

Splitting training from search depends on the company

The most useful finding for anyone writing their own robots.txt is how often sites treat the two jobs separately, and how much that varies.

Google says Google-Extended "does not impact a site's inclusion in Google Search nor is it used as a ranking signal" (Google's crawler documentation), and sites act on that. OpenAI documents its settings as independent of each other, yet half the sites that refuse GPTBot also refuse ChatGPT search, and most sites that refuse ClaudeBot also refuse Claude-SearchBot.

Across all three companies, 94 of the 214 global domains that block a training crawler (44%) allow all three AI search crawlers.

.in domains block far less

Every crawler in the study is blocked less often by the top .in domains, and the gap is widest for AI search: 2.5% block OAI-SearchBot and 3.0% block PerplexityBot, against 7.5% and 11.6% globally.

Most of those few .in blocks are not about AI at all. Of the 25 .in domains that block OAI-SearchBot, 22 do it through one wildcard rule that shuts out every unnamed crawler; only 3 name OAI-SearchBot and block it. Many .in domains also have no robots.txt we could read: 38.3%, against 29.2% globally. A domain with no robots.txt allows every crawler by default, except the few whose servers returned an error. And only 12.8% of the .in domains name any AI crawler, half the global rate of 24.7%.

The .in list reaches much further down the global ranking, to position 93,791 in the Tranco list, so it describes popular Indian brands, government portals and education sites rather than global giants. It covers India's country domain, not every Indian business; many use .com. For a business competing on .in, very few rivals are opting out of AI search today.

One wildcard rule blocks crawlers nobody decided about

46 global domains and 25 .in domains disallow the whole site for the wildcard group (User-agent: *). More than half of the global ones (25 of 46) then allow Googlebot by name, but any crawler without its own group is blocked. That single rule accounts for 31 of the 75 global OAI-SearchBot blocks and 22 of the 25 .in ones.

A related trap runs the other way. Under the robots.txt standard, a crawler that finds a group naming it follows only that group (RFC 9309), so a group that names a crawler to allow one folder can release it from every rule under the wildcard. We did not measure how often this happens, but we explain the fix in The AI Crawler Audit.

What to do with your own robots.txt

1. Paste your site into the free AI Crawler Checker. It sorts each agent into allowed, blocked and not mentioned.
2. Decide training and search separately. Blocking OAI-SearchBot keeps you out of ChatGPT search answers (a link may still appear), and blocking Claude-SearchBot or PerplexityBot can reduce or remove your visibility in those answers; blocking GPTBot or ClaudeBot does not.
3. If your file starts with User-agent: * and Disallow: /, check which AI search crawlers that rule shuts out without anyone deciding to.
4. Test the file in the robots.txt checker before you publish it, or build a new one with the robots.txt generator.

How we did it

• The domains. We took the Tranco list published on 27 October 2026 (list ID GQ3LK), a research ranking that combines five sources of popularity data. The global sample is the first 1,000 domains that answered our request for robots.txt with any HTTP response; that reaches rank 1,335, and the other 335 domains in that range did not answer (no DNS record, a refused or dropped connection, a TLS certificate error, or a timeout). The .in sample is the first 1,000 .in domains, including .co.in, .gov.in and .ac.in, that answered, out of 1,185 tried. Tranco ranks domains rather than websites, so both lists include some content delivery, advertising and API domains; we kept them rather than judge which domains count.
• The crawl. On 29 October 2026 (IST) we requested /robots.txt from each domain, then from its www address if the first failed, following redirects, with a user agent that names Veritas. A response that was an error or an HTML page counts as no robots.txt we could read. Server errors (5xx), which RFC 9309 says a crawler should treat as a full block, were also counted as no file (6 global and 15 .in domains).
• The rule. A crawler counts as blocked when the file disallows the root path (/) for it, under RFC 9309 group matching, using the same parser as our free robots.txt checker. We did not count partial blocks, and we did not check whether crawlers obey the file.
• Limits. robots.txt is a request, not an enforcement. Some sites block crawlers at the network level instead, which a robots.txt study cannot see. User-triggered fetchers such as ChatGPT-User may not follow robots.txt at all; OpenAI says robots.txt rules "may not apply" to them. Popularity lists also change daily.

Download the data

• Global top 1,000 (CSV)
• Top 1,000 .in domains (CSV)

Each row is one domain, with its Tranco rank and a yes or no for every crawler. The same files carry the llms.txt and structured data results we will publish later this month. You are welcome to use them; please cite "Veritas AI Crawler Study, October 2026" and link to this page.