Home› Site analysis› Robots.txt Tester
Robots.txt Tester
Fetch a robots.txt and test whether a path is crawlable.
About the Robots.txt Tester
Enter a domain and a path, and this fetches the live robots.txt, parses it the way Google's parser does and tells you whether that path is allowed for the bot you pick.
The rule most people get wrong is precedence. When an Allow and a Disallow both match a URL, the longer, more specific rule wins — not the first one in the file. That is why Disallow: /wp-admin/ followed by Allow: /wp-admin/admin-ajax.php works correctly, and why reordering the lines changes nothing.
Remember what robots.txt does and does not do. It stops crawling, not indexing. A blocked page that other sites link to can still appear in results as a bare URL with no description, because Google knows the page exists but has agreed not to look at it. To keep a page out of the index, allow crawling and serve a noindex directive instead.
Common questions
Which rule wins when Allow and Disallow both match?
The longer, more specific path wins, not whichever line comes first in the file — that's why reordering rules in robots.txt often changes nothing.
Does blocking a page in robots.txt remove it from Google?
No. It stops crawling, not indexing. A blocked page that other sites link to can still show up as a bare URL with no snippet, because Google knows it exists but has agreed not to look inside it.
Can I test rules before they go live?
This tool fetches the live robots.txt from the domain you enter, so test on a staging subdomain first if you want to check rules that aren't published yet.