robots.txt Tester
Paste a robots.txt and test whether a URL is blocked, with the exact rule that decided it and why that rule won.
Paste a robots.txt and test whether a given URL is actually blocked, with the exact rule that decided it.
Result
Tested
6
Blocked
1
Groups
2
Group in effect
User-agent: googlebot
A named group matched, so Googlebot ignores the wildcard group entirely. Any rule you meant to apply to every crawler has to be repeated here.
No rule in the googlebot group matches this path, so it is allowed.
No rule in the googlebot group matches this path, so it is allowed.
No rule in the googlebot group matches this path, so it is allowed.
No rule in the googlebot group matches this path, so it is allowed.
The longest matching rule wins. Disallow: /private/ is 9 characters, and Allow beats Disallow on a tie.
No rule in the googlebot group matches this path, so it is allowed.
robots.txt controls crawling, not indexing. A blocked page can still appear in results if other sites link to it, because the crawler never reads the page to see a noindex tag. To keep something out of the index, allow the crawl and use a noindex meta tag.
Crawler: Googlebot
ALLOWED /admin/settings
ALLOWED /admin/public/guide
ALLOWED /report.pdf
ALLOWED /report.pdf?v=2
BLOCKED /private/notes (Disallow: /private/)
ALLOWED /blog/postAbout the robots.txt tester
robots.txt looks simple and is not. The rule that applies is the longest matching pattern, not the first one written, and Allow beats Disallow on a tie. A file can block something you meant to allow while reading as though it does the opposite.
This implements the real matching, including wildcards and the end of path anchor, and picks the right group for the crawler you choose. For every path it shows which rule won and why.
How to use it
- 1Paste your robots.txt.
- 2List the paths you want to test.
- 3Choose the crawler, since a named group overrides the wildcard one.
- 4Read which rule decided each result.
Questions
Does robots.txt keep a page out of Google?
No, and this is the most damaging misunderstanding about it. It controls crawling, not indexing. A blocked URL can still be listed if other sites link to it, because the crawler never reads the page to find your noindex tag. To remove a page, allow the crawl and use a noindex meta tag.
Which rule wins when several match?
The longest pattern, regardless of its order in the file. If two match at the same length, Allow wins over Disallow.
What do the wildcard and dollar sign mean?
An asterisk matches any run of characters, and a dollar anchors to the end of the path. So a rule ending in .pdf and a dollar blocks report.pdf but not report.pdf with a query string, because the query comes after the end.
Why does my Googlebot group ignore the wildcard rules?
A crawler obeys only its most specific matching group. Once a Googlebot group exists, Googlebot stops reading the wildcard group entirely, so any shared rules have to be repeated.