robots.txt Generator — crawler rules without the syntax errors
Build a robots.txt for WordPress, an online store or a general site, with sitemap declaration and optional AI crawler blocking.
Free and instant — results appear in seconds. No sign-up, no limits, and nothing you type is stored.
robots.txt is a request, not a control. It tells well-behaved crawlers which paths not to fetch. Search engines honour it. Scrapers and malicious bots read it too — sometimes as a list of directories worth investigating.
The single most important thing to understand is that blocking a URL in robots.txt does not remove it from search results. Google can still index a blocked URL based on links pointing at it, and will show it with no description because it was not permitted to look. If you want a page out of the index, allow crawling and use a noindex meta tag instead. The two directives are frequently confused, and using the wrong one produces the opposite of what was intended.
Pick a preset, add your sitemap URL, and place the result at the root of your domain.
A WordPress robots.txt, annotated
User-agent: *
Disallow: /wp-admin/
Disallow: /wp-includes/
Disallow: /wp-login.php
Disallow: /?s=
Disallow: /search/
Disallow: /*?replytocom
Allow: /wp-admin/admin-ajax.phpSitemap: https://example.com/sitemap.xml
```
/wp-admin/ and /wp-includes/ are administrative paths with nothing worth crawling. Blocking them saves crawl budget.
Allow: /wp-admin/admin-ajax.php is the line people delete, and it matters. Many themes and plugins load front-end content through that endpoint. Blocking it means the crawler fetches a page missing content that real visitors see — which is exactly the situation Google's guidance on blocked resources warns about.
/?s= and /search/ block internal search result pages. These generate effectively unlimited unique URLs from crawler-followed links, all of them thin duplicates of existing content.
**/*?replytocom** blocks WordPress comment-reply links, another source of near-infinite low-value URLs. The * is a wildcard; Google and Bing support it, though it is not part of the original specification.
The Sitemap line is independent of any user-agent block and can appear anywhere in the file. It is the simplest way to tell crawlers where your sitemap lives without registering anything.
Note what is not here: no attempt to block /wp-content/uploads/. Blocking it hides your images from image search for no benefit.
robots.txt does not remove pages from search
This is worth stating twice because getting it backwards is common and the consequences are silent.
Disallow prevents fetching. It does not prevent indexing. A blocked URL that other pages link to can still appear in results, listed as a bare URL with no snippet — because the crawler was forbidden from looking at it.
Worse, blocking a page that already carries a noindex tag guarantees it stays indexed, because the crawler can no longer fetch the page to see the tag.
The correct approach:
- Keep it out of the index → allow crawling, add
<meta name="robots" content="noindex">. - Save crawl budget on worthless URLs →
Disallowin robots.txt. - Genuinely private → require authentication. robots.txt is a public file and offers no protection.
File placement
It must sit at the root of the domain: https://example.com/robots.txt. A file in a subdirectory is ignored entirely.
Each subdomain needs its own. blog.example.com does not inherit the rules from example.com. Protocol and port also define separate origins for this purpose.
Crawl-delay
Google ignores Crawl-delay completely. Crawl rate for Googlebot is adjusted in Search Console. Bing and Yandex do honour the directive.
If a crawler is genuinely overloading your server, rate limiting at the server or CDN is the reliable answer, since anything that ignores robots.txt will ignore the delay too.
Blocking AI crawlers
Several operators publish named user agents for training data collection — GPTBot, CCBot, ClaudeBot, Google-Extended, PerplexityBot and others. Blocking them is a policy decision, not a technical one.
Two things to weigh. It relies entirely on voluntary compliance; anything that disregards robots.txt is unaffected. And some of these agents also feed retrieval features that cite and link to sources, so blocking them can remove your site from answers that would otherwise have sent traffic. Google-Extended in particular controls training use without affecting Search indexing — the two are separate.
Mistakes that cost traffic
Disallow: / left on a live site. Deployed from a staging configuration, this removes the entire site from search. It is the single most expensive one-character mistake in SEO.
Blocking CSS and JavaScript. Search engines render pages. Blocked assets mean a broken render and misjudged mobile usability.
Assuming it is private. Anyone can read yoursite.com/robots.txt. Listing /admin-secret/ advertises it.
Trailing slash confusion. Disallow: /private blocks /private-files.html as well as /private/. Add the slash if you mean the directory.
Frequently asked questions
Will robots.txt remove a page from Google?
No. It prevents crawling, not indexing. A blocked page can still appear in results as a bare URL with no description if other pages link to it. To remove a page from the index, allow crawling and add a noindex meta tag — blocking it actually prevents Google from ever seeing that tag.
Where does robots.txt go?
At the root of the domain, reachable at https://example.com/robots.txt. In a subdirectory it is ignored completely. Each subdomain needs its own file; blog.example.com does not inherit rules from example.com.
Should I block AI crawlers?
That is a policy decision rather than a technical one. Blocking relies on voluntary compliance, so anything ignoring robots.txt is unaffected. Note also that some of these agents power retrieval features that cite and link sources, so blocking them can remove you from answers that would have sent traffic. Google-Extended controls training use only and does not affect Search indexing.
Does Google respect Crawl-delay?
No. Googlebot ignores the directive entirely; crawl rate is adjusted through Search Console instead. Bing and Yandex do honour it. If a crawler is genuinely overloading your server, rate limiting at the server or CDN is more reliable, since anything ignoring robots.txt will ignore the delay too.
Can I hide private files with robots.txt?
No, and attempting it makes matters worse. The file is publicly readable, so listing a path advertises its existence to anyone curious. Genuinely private content needs authentication or server-level access control.
Do I need to block wp-content/uploads?
No, and you should not. That directory holds your images, and blocking it removes them from image search with no compensating benefit. The paths worth blocking on WordPress are the admin directories and parameter-driven URLs such as internal search results.
Reviews
No reviews yet. If this tool solved something for you, yours would be the first — and it helps other people decide whether it is worth their time.