robots.txt Tester
Check whether a crawler is allowed to fetch a specific URL, using the same rules the crawler applies: the matching user-agent group, wildcards, and the longest matching pattern.
About
A robots.txt file is a set of politely obeyed instructions, and the rules that decide whether a URL may be fetched are more subtle than they look: the matching user-agent group, wildcards, and the rule with the longest matching pattern. This tool applies those rules the way a crawler does and shows which line decided the outcome.
How to use
- Enter the site and the path you want to test.
- Pick the crawler you care about, or leave it as Googlebot.
- Submit to see the verdict and the rule that produced it.
Frequently asked questions
Which rule wins when several match?
The longest matching pattern. If two rules are the same length, the allow rule wins. That is why a short Disallow can be carved out with a longer Allow.
Does robots.txt keep a page out of search results?
Not reliably. A blocked URL can still be indexed from links if it is not also marked noindex. Blocking and noindex together are also a trap, because a blocked page cannot be read to see the noindex.
What does the asterisk do?
It matches any sequence of characters, and a trailing dollar sign anchors the pattern to the end of the URL. Without the anchor, a pattern matches any path that starts with it.
Is robots.txt a security measure?
No. It is a request that well-behaved crawlers honour. Anything private needs authentication, because any client can simply ignore the file.