robots.txt Generator
Write a robots.txt that lets answer engines in and keeps training crawlers out, in two clicks.
No sign-up. We read the public file for you. Nothing is stored.
What you get
- A file you can paste straight into your site root
- Which AI crawlers it lets in, by name
- Whether Google can still crawl you
- What your current robots.txt does today
- Writes a separate group for each AI crawler you block, because a crawler that has its own group stops reading your wildcard rules.
- Keeps the answer engines that cite you (OAI-SearchBot, PerplexityBot, Claude-SearchBot and the rest) separate from the crawlers that only collect training data.
- Reads the file back through a real robots.txt parser and reports what a crawler would conclude, including whether Googlebot can still reach you.
- Drops a Sitemap line that is not a full https:// URL rather than publishing one that no crawler will follow.
How to use the robots.txt Generator
Enter your domain
We read the robots.txt you have now, so you can see what is about to change. No file yet is fine.
Make the two decisions
Whether answer engines may read you, and whether training crawlers may. Then add any paths to keep crawlers out of.
Copy it to your site root
It has to answer at yourdomain.com/robots.txt. A crawler looks nowhere else.
Why write robots.txt by hand at all?
One line can remove you from search.
A stray Disallow: / on the wrong group takes a site out of Google entirely, and nothing on the page will tell you it happened.
Naming a crawler changes the rules.
A crawler obeys the most specific group that names it and ignores your * rules from then on, so a block written under the wildcard blocks nobody.
Answers and training are separate choices.
Keeping your words out of a training set costs you nothing. Blocking the crawler that fetches pages to answer a question costs you the citation.
Most generators ignore AI crawlers.
They emit a wildcard and a sitemap line, which is exactly the file that leaves every decision about answer engines unmade.
What to do with the result
A robots.txt is the shortest file on your site and the one with the most expensive mistakes: it decides whether Google can index you and whether an answer engine can quote you. The version this writes allows the crawlers that put you in answers, turns away the ones that only harvest training data, and says which pages are none of anyone’s business. Once the file says what you meant, the question worth asking is whether those engines are citing you, which is what AI Visibility Tracking measures.
AI Visibility TrackingFrequently asked questions
At the root of the domain, so it answers at https://yourdomain.com/robots.txt. A crawler looks there and nowhere else, so a file in a subfolder does nothing. It also applies per host: www and the bare domain are separate, as is every subdomain.
More free tools
AI Crawler Checker
See exactly which AI crawlers you allow, block, or never mentioned, from just your domain.
llms.txt Validator
Check any domain’s llms.txt for the mistakes that stop an assistant using it properly.
SERP Snippet Preview
See how your title and description render in Google results, and where they get cut off.
See where you actually stand.
A free tool checks one thing once. Veritas watches all of it, continuously, across every engine that decides whether you get cited.
GOOGLE · CHATGPT · PERPLEXITY · GEMINI · AI OVERVIEWS