Robots.txt Tester
Enter a domain and a path, and this tool fetches that site's live robots.txt, then tells you exactly whether the crawler you pick is allowed to request that URL — and which rule made the decision.
What Is Robots.txt Tester?
A robots.txt tester answers one specific question: for this URL, on this site, is this particular crawler allowed to request it? That's a narrower question than "is robots.txt valid," which is what a validator checks instead — a robots.txt file can be perfectly valid syntax and still block a URL you didn't intend to block.
This distinction matters because robots.txt rules are evaluated per user-agent, and the results genuinely can differ between crawlers. A site might disallow GPTBot entirely while allowing Googlebot full access, or block an image crawler from a folder that the main search crawler can still reach. Testing against the specific agent you care about is the only way to get an answer that's actually relevant to your situation.
How to Use This Tool
- 1Enter the domain you want to check (just the root domain is enough — the tool fetches /robots.txt from it automatically).
- 2Enter the specific path you're testing, starting with a slash, like /blog/some-post or /category/widgets.
- 3Choose which crawler's rules to evaluate against — Googlebot and the wildcard (*) group are the most common checks.
- 4Review the result: allowed or blocked, plus the exact rule and rule group that produced that decision.
Understanding Your Results
"Allowed" or "Blocked"
This is the direct answer to whether the chosen crawler can request the path you entered, based purely on robots.txt rules. It says nothing about whether the page is actually indexed, ranking, or reachable for other reasons — robots.txt is only one of several signals that affect crawling and indexing.
Matched rule and rule group
Robots.txt evaluates the most specific matching rule, not simply the first one that appears in the file. This tool shows you exactly which Allow or Disallow line decided the outcome, and which User-agent group it belongs to, so you're not left guessing why a particular result came back.
No robots.txt found
If a site has no robots.txt file at all, the standard interpretation is that crawling is fully allowed — there are no rules to restrict it. This is different from an empty robots.txt file, or one that returns a server error, both of which crawlers may interpret differently depending on the specific crawler's implementation.
Common Problems & How to Fix Them
A whole section blocked by accident
A broad rule like Disallow: /blog can unintentionally block far more than intended if a more specific Allow rule wasn't added for the parts that should stay crawlable. Test a few representative URLs from the section in question, not just the top-level path, since a folder can be disallowed while a specific important page inside it was meant to stay open.
Rules that only apply to one crawler
It's common to disallow a bot like an AI crawler while leaving Googlebot's group untouched, and just as common to accidentally place a rule in the wrong User-agent group, causing it to apply more narrowly (or broadly) than intended. Always test against the specific crawler you actually care about rather than assuming the wildcard group covers every bot.
SEO Best Practices
Test your most important pages — homepage, key category pages, and any page you've recently changed robots rules for — after any robots.txt edit, rather than assuming the change did what you intended.
Remember that robots.txt controls crawling, not indexing. A blocked page can sometimes still appear in search results (typically without a snippet) if other sites link to it. Use a noindex directive if the real goal is keeping a page out of search results entirely.
Keep rules as specific as practical. Broad wildcard disallows are easy to write and easy to get wrong; a handful of precise rules is safer than one broad one you have to keep testing against every URL pattern on the site.
Frequently Asked Questions
Does this tool check Google's actual crawling behavior?
No — it evaluates the robots.txt rules exactly as written, the same way a compliant crawler would. It doesn't query Google directly, since Google doesn't expose a live API for that. Well-behaved crawlers follow these rules, though not every bot on the internet respects robots.txt.
Why does the same path show different results for different crawlers?
Because robots.txt rules are defined per User-agent group, and sites often intentionally treat crawlers differently — for example, allowing search engines while restricting AI training crawlers.
What does it mean if no rule matches my path at all?
If nothing in the relevant User-agent group's rules matches, crawling defaults to allowed for that path — robots.txt is a blocklist model, not an allowlist model, unless an Allow rule explicitly says otherwise.
Can I test a path that doesn't exist yet?
Yes — robots.txt matching works on the path pattern itself, not on whether a page actually exists there. This is useful for confirming rules before a new section of the site goes live.