AI Crawler Checker
See exactly which AI crawlers you allow, block, or never mentioned, from just your domain.
No sign-up. We read the public file for you. Nothing is stored.
What you get
- Which AI crawlers you allow
- Which you block, deliberately or not
- Which you never mentioned at all
- The exact rule responsible for each verdict
- Which AI crawlers your robots.txt allows, which it blocks, and which it never mentions at all.
- GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, PerplexityBot, Google-Extended and the other agents that decide whether you can be cited.
- The group-precedence trap: naming a user-agent gives it its own block, so it stops obeying your wildcard rules entirely.
- Blanket disallow rules meant for scrapers that catch answer-engine crawlers as collateral.
How to use the AI Crawler Checker
Enter your domain
We read the robots.txt at its root for you. Staging site or a draft? Paste it instead.
Check crawlers
One click. Sixteen agents, grouped into the ones that cite you and the ones that train on you.
Read the three columns
Allowed, blocked, and unmentioned. The third column is usually the surprising one.
Why check which AI crawlers you allow?
Blocking removes you from the answer.
If GPTBot or PerplexityBot cannot read the page, you cannot be cited on it. There is no second route in.
Most blocks were never chosen.
A security plugin, a CDN default, or a wildcard written for scrapers years ago. Very few teams decided to block; many are blocking.
Naming a bot changes everything.
Give a user-agent its own group and it stops obeying your wildcard rules entirely. This is the trap that catches careful people.
Training and search are separable.
Blocking training while allowing search is a legitimate middle position, and it only works if the agents are listed separately.
What to do with the result
Blocking an AI crawler removes you from the answers it writes, and most sites that block one never chose to: a security plugin, a CDN rule, or a wildcard written years ago did it for them. Whether to allow them is a business decision rather than a technical default, but it should be a decision. Once the file says what you meant, the next question is whether the engines are actually citing you, which is what AI Visibility Tracking measures.
AI visibility toolEmbed this tool
Paste this into any page. It loads the AI Crawler Checker without our navigation, with a one-line credit under it. Free to use on agency sites, client reports and course pages. Keep the credit line.
<iframe src="https://www.getveritas.io/embed/ai-crawler-checker" width="100%" height="980" style="border:0;max-width:100%" title="AI Crawler Checker by Veritas" loading="lazy"></iframe>
<p style="font:13px/1.5 system-ui,sans-serif;margin:6px 0 0">Free <a href="https://www.getveritas.io/tools/ai-crawler-checker">AI Crawler Checker</a> by Veritas</p>Frequently asked questions
Two common ways. A blanket Disallow rule meant for scrapers catches them, or a security plugin or CDN adds bot rules you never saw. The subtler one: naming a user-agent gives it its own group, and it then stops obeying your * rules entirely, so a rule you thought applied to everyone silently does not apply to the bot you named.
More free tools
llms.txt Validator
Check any domain’s llms.txt for the mistakes that stop an assistant using it properly.
llms.txt from Sitemap
Turn any domain’s sitemap into a first-draft llms.txt, grouped into sections automatically.
Open Graph Checker
See how your page looks when it is shared on LinkedIn, X, Facebook and Slack, and get the tags to fix it.
See where you actually stand.
A free tool checks one thing once. Veritas watches all of it, continuously, across every engine that decides whether you get cited.
GOOGLE · CHATGPT · PERPLEXITY · GEMINI · AI OVERVIEWS