robots.txt Checker
Test any URL against any crawler, see the exact rule that decides it, and find the lines crawlers ignore.
No sign-up. We read the public file for you. Nothing is stored.
Edits stay in this tab. Nothing is uploaded or changed on your site.
What you get
- Whether a URL is allowed or blocked
- The exact rule and group that decided it
- Nine common crawlers checked at once
- Errors, typos and ignored directives
- Whether a URL may be fetched by a given crawler, using the same precedence crawlers apply: the most specific group, then the longest matching rule, with Allow winning ties.
- Which rule and which user-agent group decided the answer, so you can see exactly what to change.
- The same URL against Googlebot, Bingbot, OAI-SearchBot, GPTBot, PerplexityBot, ClaudeBot, Google-Extended, Applebot and CCBot at once.
- Lines crawlers skip or misread: misspelt directives, rules before any User-agent, unsupported noindex, relative Sitemap URLs and files over Google’s 500 KiB limit.
How to use the robots.txt Checker
Enter your domain
We fetch the live robots.txt from the root of the domain. No file is a valid answer: it means everything may be crawled.
Test the URLs that matter
Type a path or a full URL and pick a crawler. The verdict names the rule that decided it.
Edit and retest before publishing
Change the file in the editor and every result updates. Publish the version that gives the answers you want.
Why test robots.txt rather than read it
Precedence is not top to bottom.
The longest matching rule wins, Allow wins a tie, and a crawler with its own group ignores the wildcard group entirely. Reading the file in order gives the wrong answer surprisingly often.
Mistakes fail silently.
A misspelt directive, a rule before any User-agent line, or a noindex line Google stopped supporting in 2019 is simply skipped. Nothing reports it.
One line can hide a whole site.
Disallow: / under the wrong group removes every page from search, and the site keeps working for everyone else.
Editing is safer when tested.
Change the file here, test the URLs that matter, and publish only when every answer is the one you intended.
What to do with the result
robots.txt is short and unforgiving: the precedence rules are not intuitive, mistakes are silent, and a single line can take a site out of search. Testing the URLs that matter against the crawlers that matter is the only reliable way to know what the file does. Once it says what you meant, Site Audit checks that the pages you allowed are actually healthy for Google and for AI crawlers.
Site AuditFrequently asked questions
Enter your domain, type the URL or its path, and choose the crawler. The checker applies the same rules crawlers do and names the line that decides the answer. Google retired its standalone robots.txt Tester in 2023; Search Console now shows a robots.txt report instead.
More free tools
Sitemap Validator
Check an XML sitemap for the problems that make crawlers skip URLs: duplicates, other hosts, http and bad dates.
Redirect Checker
Trace every redirect a URL makes: the status code, where it points, and how long each hop takes.
Schema Markup Generator
Generate valid JSON-LD for sixteen schema.org types, from Article and Product to Event, Job Posting and Review.
See where you actually stand.
A free tool checks one thing once. Veritas watches all of it, continuously, across every engine that decides whether you get cited.
GOOGLE · CHATGPT · PERPLEXITY · GEMINI · AI OVERVIEWS