robots.txt

robots.txt is the oldest control a site owner has over automated visitors. It lives at the top level of a host, is organised into groups that each start with a User-agent line, and lists Allow and Disallow paths for the agents that group names. A crawler reads the group that names its own token, or the wildcard group if none does.

The format is now a published standard, RFC 9309, which pinned down details that used to vary between crawlers, such as how groups are matched and how much of the file a crawler must read.

AI search has made this small file strategic again. The same few lines now decide whether a page can be used to train a model, whether it can be indexed for an assistant's answers, and whether it is crawled for classic search, and those are separate decisions with separate user agents.

It is also widely misunderstood. A disallow line is a request that compliant crawlers honour. It is not access control, it does not reliably keep a URL out of search results, and some user-triggered fetchers may not apply it at all.

• RFC 9309, published in September 2022, says the rules "MUST be accessible in a file named '/robots.txt' (all lowercase) in the top-level path of the service", that "These rules are not a form of access authorization", and that crawlers must parse at least 500 kibibytes of the file.IETF, RFC 9309: Robots Exclusion Protocol, 15 Sept 2026
• Google says robots.txt "is not a mechanism for keeping a web page out of Google", and that to do that you should "block indexing with noindex or password-protect the page".Google Search Central, Introduction to robots.txt, 15 Sept 2026
• Google says "Google-Extended does not impact a site's inclusion in Google Search nor is it used as a ranking signal in Google Search"; it manages whether crawled content may be used for training future Gemini models.Google Search Central, Google's common crawlers, 15 Sept 2026
• OpenAI says "it can take ~24 hours from a site's robots.txt update for our systems to adjust" for search results, and that for ChatGPT-User, "Because these actions are initiated by a user, robots.txt rules may not apply."OpenAI, Overview of OpenAI crawlers, 15 Sept 2026

1. 1

   Serve the file at the root of each host and protocol you run, as /robots.txt in lowercase. A subdomain needs its own file.

2. 2

   Keep a wildcard group (User-agent: *) that states your default, then add a group only for agents you want to treat differently.

3. 3

   Name AI agents by their documented tokens. Training: GPTBot, ClaudeBot. AI search: OAI-SearchBot, Claude-SearchBot, PerplexityBot. For Gemini training, use the Google-Extended token; it does not change how Googlebot crawls you for Search.

4. 4

   Use noindex, not Disallow, for pages that must not appear in search. A disallowed URL cannot be crawled, so a noindex on it is never seen.

5. 5

   After any change, fetch the live file and confirm which group each agent lands in, then allow time for vendors to pick it up.

• Disallowing a page to hide it, which can leave the bare URL visible in results while blocking the crawler from ever reading a noindex.
• Blocking all AI user agents to opt out of training, which also blocks the search agents that decide whether assistants can cite you.
• Assuming blocking Google-Extended removes a site from AI Overviews. Google documents it as a training control that does not affect inclusion in Google Search.
• Using robots.txt to protect private content. It is published, readable by anyone, and advisory.

noindex

robots.txt controls whether a URL may be fetched. noindex, set in a meta tag or header, controls whether a fetched page may appear in search results, and only works if the page can be crawled.

llms.txt

robots.txt restricts what crawlers may fetch. llms.txt is a proposed markdown file that points language models at the content worth reading, and grants or blocks nothing.

1. 1IETF, RFC 9309: Robots Exclusion Protocol read 15 Sept 2026
2. 2Google Search Central, Introduction to robots.txt read 15 Sept 2026
3. 3Google Search Central, Google's common crawlers read 15 Sept 2026
4. 4OpenAI, Overview of OpenAI crawlers read 15 Sept 2026