Robots.txt Tester
Fetch a live robots.txt and test whether a specific crawler is allowed to reach a specific path.
Free and instant — results appear in seconds. No sign-up, no limits, and nothing you type is stored.
Enter a domain and a path, choose a crawler, and this fetches the live robots.txt and tells you whether that path is allowed — along with the exact rule that decided it.
Rules interact in ways that are easy to misread. When Disallow: /blog and Allow: /blog/public both match, the longer path wins, and that is not obvious from reading the file.
A worked example
Testing /wp-admin/admin-ajax.php for Googlebot on a typical WordPress site:
The file
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.phpSitemap: https://example.com/sitemap.xml
```
Result: allowed
Matched by Allow: /wp-admin/admin-ajax.php. Both rules match the path, but the Allow rule is longer and therefore more specific, so it wins.
This pairing is deliberate and correct. WordPress themes call admin-ajax.php for front-end functionality, so blocking it can stop Google rendering pages properly — while the rest of /wp-admin/ genuinely should not be crawled.
A variation that breaks
Add a Googlebot-specific group and things change:
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.phpUser-agent: Googlebot
Disallow: /private/
```
Googlebot now reads only the second group. The admin-ajax.php allowance no longer applies to it — but neither does the /wp-admin/ block, so Googlebot is free to crawl the whole admin directory.
The author almost certainly intended to add one rule. They replaced the entire ruleset instead.
Disallow does not mean noindex
This is the single most expensive misunderstanding in the file's history, so it goes first.
Disallow prevents crawling. It does not prevent indexing. If other sites link to a blocked URL, Google can and does index it using the anchor text of those links — producing a search result with a URL, no description, and the note that no information is available.
Worse, blocking a page in robots.txt makes a noindex tag on that page unreachable. The crawler never fetches the page, never sees the tag, and never learns you wanted it excluded.
To keep something out of the index: allow crawling and use noindex. To keep something private: use authentication. robots.txt is neither of those things — it is a public file suggesting what to crawl.
Longest match wins
Where rules conflict, the most specific path takes precedence, regardless of order in the file. Allow: /blog/public/post beats Disallow: /blog because it is longer. This tool shows which rule matched so you can see the resolution rather than guess at it.
User-agent groups do not merge
A crawler obeys exactly one group — the most specific one naming it. If there is a User-agent: Googlebot block, Googlebot reads that and completely ignores User-agent: *, even for rules that only appear in the wildcard block.
This catches people constantly. Adding a Googlebot-specific rule silently discards every wildcard rule for Googlebot. Anything that should apply must be repeated in the specific group.
AI crawlers
GPTBot, ClaudeBot, CCBot and others are now common entries. They are honoured on the same voluntary basis as everything else in the file: a well-behaved crawler respects it, and a badly-behaved one was never going to.
Blocking them is a legitimate publishing decision. Just be aware it is a request, not a control, and it has no bearing on search rankings either way.
Frequently asked questions
Will Disallow remove a page from Google?
No, and this is the most common misconception about the file. Disallow stops crawling, not indexing. A blocked URL can still appear in results if other pages link to it — with no description, because Google never fetched it. To remove a page, allow crawling and use a noindex tag.
Can I use noindex in robots.txt?
No. Google supported an undocumented noindex directive for years, then stopped honouring it in September 2019. If you still have one, it does nothing. Use a meta robots tag or an X-Robots-Tag header on the page itself.
Does robots.txt keep content private?
The opposite. It is a public file at a predictable location, so listing a path there advertises it to anyone curious. Private content needs authentication. Sensitive paths should never be named in robots.txt.
Why is my page blocked when I did not block it?
Usually a wildcard catching more than intended, or a user-agent group problem. A specific group for your crawler makes it ignore the wildcard group entirely, so rules you assumed applied may not. This tool shows the exact matching rule, which normally settles it.
Should I block AI crawlers like GPTBot?
That is a publishing decision, not an SEO one — it has no effect on rankings either way. Blocking is honoured voluntarily by crawlers that choose to comply. If your content is a commercial asset you would rather not see used for training, blocking is reasonable. If discovery matters more, allow them.
Do I need a robots.txt file?
No. Without one, everything is crawlable, which is a perfectly valid default. A missing file is not an error and this tool reports it as such. Add one when you actually have something to exclude, or to declare your sitemap location.
Are the rules case-sensitive?
Paths are, directive names are not. Disallow: /Blog will not match /blog. This trips people up on servers that treat URLs case-insensitively, where both spellings resolve but only one is blocked.
Reviews
No reviews yet. If this tool solved something for you, yours would be the first — and it helps other people decide whether it is worth their time.