Robots.txt Tester
Compare a URL path with robots.txt Allow/Disallow rules using RFC 9309 group selection, percent-encoding rules and longest-match precedence, with a separate Google-style full User-Agent mode.
Matching uses path + query. URL fragments are ignored because they are not sent in HTTP requests.
Selected crawler group, matching Allow/Disallow rules, winning specificity and crawl decision will appear here.
The First Question Is “Which Group Applies?”
RFC 9309 describes crawlers using a product token such as ExampleBot. Matching that token is case-insensitive, and repeated groups with the same matching token are combined. A User-agent: * group is only the fallback when no specific group matches.
Google also documents how it selects the most specific product token from its crawler's complete HTTP User-Agent identification string. The two input modes make that distinction explicit instead of silently treating a full UA string as a product token.
Multiple Matching Groups Are Merged; the Wildcard Group Is Not Added on Top
Two separate User-agent: Googlebot groups contribute rules to the same effective group. But once that specific group exists, User-agent: * is not combined with it.
This matters when a global Disallow appears stricter than the crawler-specific group. Merging the wildcard rules on top of a specific group can falsely block a URL that the crawler would actually allow.
Longest Matching Rule Wins—and Equivalent Allow Is Preferred
A crawler evaluates all matching Allow and Disallow patterns in the selected group, then uses the most specific match. RFC 9309 defines specificity in octets. If equivalent Allow and Disallow rules tie, Allow should win.
Percent-encoded bytes therefore cannot be compared reliably by simply counting JavaScript characters. Unreserved escapes are canonicalized before matching, while a remaining %HHtriplet represents one encoded byte in the normalized path.
* and $ Are Pattern Operators, Not Regular Expressions
Robots path syntax has only a small wildcard vocabulary: * matches zero or more characters and a final $ anchors the end. Google documents trailing * as redundant, so /fish* behaves like /fish.
Characters such as ., +, parentheses and square brackets are literal URL characters here, not regex syntax. RFC 9309 contains a small inconsistency here: its ABNF starts a path-pattern with /, while its own example uses *.gif$. Reported erratum 7995 proposes allowing a leading *. That spelling is accepted for interoperability, but /*.gif$ is clearer for Google-oriented files.
Query Strings Can Change Which Rule Wins
Robots matching uses the path representation that can include the query component. An end-anchored rule such as /*.pdf$ matches a URL ending in .pdf, but not necessarily the same path followed by ?download=1.
robots.txt Is Public Crawl Guidance, Not Access Control
Disallow tells compliant crawlers not to fetch a path. It does not stop a person, browser or non-compliant bot from requesting it, and publishing a sensitive pathname in robots.txt can make that pathname easier to discover.
Use authentication/authorization for private content. Use page or response indexing controls when the actual problem is indexing rather than crawling.
Pasted Text Cannot Tell You How /robots.txt Was Served
RFC 9309 expects the file at lowercase top-level /robots.txt, served as UTF-8 text. Fetch status also matters: unavailable 4xx behavior differs from unreachable 5xx/network behavior, redirects can be followed, and crawlers can cache the result.
Pasted content also cannot prove the deployment path, HTTP status, Content-Type, cache state or the original raw byte encoding before the browser produced Unicode text. Google documents a 500 KiB parsing limit; Google-style mode applies that limit to the UTF-8 bytes of the pasted text, while generic RFC product-token mode keeps the full text and warns that another crawler may stop earlier.
Defines product tokens, group merging, longest-match behavior, encoding rules, /robots.txt access and the 500 KiB minimum parsing-support limit.
Documents Google's crawler-token selection, wildcard examples, trailing wildcard behavior and rule precedence.