Robots.txt Validator
Validate RFC 9309 robots.txt groups, rules, tokens, conflicts, Sitemap records, and deployment path assumptions.
This is used only to review the required top-level /robots.txt location and compare Sitemap origins. No URL is fetched.
RFC 9309 group structure, path-pattern findings, merged user-agent groups, Sitemap records and extension notes will appear here.
robots.txt Is Parsed as Groups and Rules, Not as “Last Line Wins”
Each standard group starts with one or more User-agent product tokens and contains Allow/Disallow rules. If the same crawler product token appears in more than one group, RFC 9309 requires the matching rule sets to be combined.
That means adding a second Googlebot or ExampleBot block lower in the file does not erase the first block. A merged-token view exposes the combined rule set instead of implying “last line wins.”
The Most Specific Matching Pattern Wins; Equal Allow Beats Equal Disallow
User-agent: * Disallow: /private/ Allow: /private/public-guide
The broader Disallow does not necessarily make every descendant inaccessible to the crawler. A more-specific matching Allow can reopen a path. When an Allow and Disallow match are equally specific, RFC 9309 says the Allow rule should be used.
This is why Allow: / and Disallow: / together do not mean “block everything.” Their specificity ties, so Allow wins.
* and Terminal $ Are Pattern Operators
RFC 9309 gives * wildcard meaning and uses a trailing $ to anchor a match to the end. Those characters are therefore not ordinary literal path bytes when used in their special positions.
Matching also involves percent-encoded URI octets. A malformed % escape can make a rule difficult to reason about, so malformed escapes are reported instead of assuming the visible text has obvious request-path semantics.
A Raw # Starts a Comment—It Does Not Describe a URL Fragment Rule
In robots.txt, # introduces a comment. Separately, browser URL fragments such as #reviews are not sent to the web server as part of the HTTP request target. So trying to manage an in-page fragment with Disallow: /page#reviews is wrong in two different ways.
If a literal hash octet somehow belongs in the request-path data, URI percent-encoding is the relevant representation—not a raw comment marker.
/robots.txt Is a Service-Level Location
RFC 9309 defines discovery at the lowercase top-level path /robots.txt. Rules apply to the service identified by the scheme, host and port used to retrieve that file. A robots file buried at /seo/robots.txt is not the standard file for the whole site.
The optional HTTP(S) deployment URL exists because valid content published at the wrong location is still operationally ineffective.
Sitemap and Crawl-delay Are Not Equivalent Kinds of robots.txt Record
RFC 9309 standardizes User-agent, Allow and Disallow group parsing and explicitly permits crawlers to interpret additional records without letting them interfere with standard parsing. Sitemap: is widely supported by search engines as a sitemap-discovery record.
Crawl-delay is crawler-specific rather than an RFC 9309 standard directive. Supporting it in one bot does not make it portable to another. Such records are reported as extensions, not as universal robots syntax.
robots.txt Is Public Crawl Guidance, Not Access Control
The file itself is publicly retrievable and often advertises the paths you are trying to discourage crawlers from fetching. Compliant crawler behavior is not authentication. Sensitive admin, account, document or API data needs real authorization.
Blocking crawl also does not guarantee a URL disappears from search results if the URL is discovered from other sources. Crawling, indexing and authorization are separate controls.
Pasted Syntax Cannot Tell You What a Live Crawler Actually Retrieved
The live file can return 404, 5xx, redirect, be cached, be served with unexpected bytes, or differ between CDN and origin. RFC 9309 defines crawler behavior around successful retrieval, redirects, unavailable files and caching; none of those facts exist in pasted text.
After syntax review, test the deployed URL and use the target search engine's debugging tools when the question is “what did this crawler actually see?”
/robots.txtlocation. For search-engine-specific records and behavior, use that crawler's own current documentation.