WordPress robots.txt Generator
Check the robots.txt your WordPress site serves, then copy the file it should: open to Google, closed to the admin, and clear about AI crawlers.
No sign-up. We read the public file for you. Nothing is stored.
Updated
What you get
- What your current robots.txt blocks
- A recommended file, ready to paste
- Proof that theme files and uploads stay open
- Where to paste it in Yoast, Rank Math or AIOSEO
- Whether your current file blocks the homepage, uploads, theme CSS, core JavaScript or admin-ajax.php, judged the way Googlebot reads it.
- Which sitemap your site declares, so the new file points at the one your SEO plugin actually serves: wp-sitemap.xml, sitemap_index.xml or sitemap.xml.
- Leftovers from copied templates: blocked feeds and archives, and noindex or Crawl-delay lines Google ignores.
- That the file you copy leaves every path WordPress needs open, read back with the same rules Googlebot uses.
How to use the WordPress robots.txt Generator
Enter your WordPress domain
We read the robots.txt it serves now and check what it blocks. No file is fine; WordPress normally serves a virtual one.
Choose a few options
Your sitemap plugin, whether you run WooCommerce, and what AI crawlers may do. Sensible defaults are already set.
Paste it into your SEO plugin
Yoast, Rank Math and All in One SEO each have a robots.txt editor. Without a plugin, upload the file to your site’s root folder.
Why WordPress sites get robots.txt wrong
Copied files block the theme.
Old “SEO” templates disallow /wp-content/ and /wp-includes/. That hides the CSS and JavaScript Google needs to render the page, so it sees a broken layout.
admin-ajax.php has to stay open.
Themes and plugins call it from public pages. WordPress’s own default allows it for that reason, and many hand-written files forget.
Search pages multiply.
Every internal search creates a new URL. Blocking /?s= keeps crawlers on pages worth indexing.
AI crawlers are a separate decision.
Training crawlers and answer engines are different bots. Blocking the first costs nothing; blocking the second stops ChatGPT and Perplexity citing you.
What to do with the result
The right robots.txt for WordPress is short: keep crawlers out of the admin, leave everything Google needs to render the site open, and say where the sitemap is. Most of the damage comes from adding more than that. Once the file is right, the question is whether the pages it leaves open are healthy, which is what Site Audit checks for Google and for AI crawlers.
Site AuditFrequently asked questions
With no file of its own, WordPress serves a virtual one: “User-agent: *”, “Disallow: /wp-admin/”, “Allow: /wp-admin/admin-ajax.php”, and since version 5.5 a Sitemap line pointing at /wp-sitemap.xml. It is a sound starting point.
More free tools
Domain Age Checker
See when any domain was registered, when it expires, who registered it, and when a website first appeared on it.
Indexability Checker
Can Google index this page? One answer, from every place that decides it: status, redirects, robots.txt, noindex and canonical.
FAQ Schema Generator
Turn a page’s questions and answers into valid FAQPage JSON-LD.
See where you actually stand.
A free tool checks one thing once. Veritas watches all of it, continuously, across every engine that decides whether you get cited.
GOOGLE · CHATGPT · PERPLEXITY · GEMINI · AI OVERVIEWS