Robots.txt Generator
Build RFC 9309-style crawler groups, validate product tokens and patterns, and keep Sitemap records separate.
Used only to show the expected top-level robots.txt location and review sitemap origins. Nothing is fetched.
Rule Group 1
Matching groups for the same product token are combined by RFC 9309 crawlers; they are not override blocks.
One token per line. RFC 9309 product tokens use letters, underscore or hyphen; * is the fallback wildcard.
One path pattern per line. Patterns begin with /. RFC 9309 supports * and terminal $; a literal # must be percent-encoded because # starts a comment.
One absolute HTTP(S) sitemap URL per line. Sitemap is a commonly supported additional robots.txt record, not an Allow/Disallow rule.
Generated crawler groups and review notes will appear here.
robots.txt Answers “May This Crawler Fetch This Path?”—Not “Is This Page Private?”
A robots.txt file is public and its rules are voluntary instructions for compliant crawlers. Anyone can request the file and see the paths you listed. A blocked URL can also be discovered through links or other sources even when a crawler does not fetch its contents.
Protect private content with authentication and authorization. Use crawler directives for crawl behavior, not as a substitute for access control.
Two Groups for the Same Bot Are Combined, Not Applied Top-to-Bottom Like CSS
User-agent: ExampleBot Disallow: /private/ User-agent: ExampleBot Allow: /private/public/
RFC 9309 requires matching groups for the same product token to be combined before their rules are evaluated. The second block does not erase the first. A repeated product token is therefore flagged when it is spread across several groups.
The Most Specific Matching Path Wins; Equal Allow Beats Equal Disallow
With Disallow: /private/ and Allow: /private/public/, a compliant crawler can fetch the more-specific public path while the broader private subtree stays disallowed. Specificity is based on the length of the matched path pattern in octets after the protocol's URI normalization rules.
If an Allow and Disallow rule are equally specific, RFC 9309 says the Allow rule should be used. That makes Allow: / plus Disallow: / a poor way to express “block everything.”
# Is a Comment Marker, So a Literal Hash Must Not Be Pasted Raw Into a Rule
In robots.txt, # starts a comment. A line intended as Disallow: /file#part would not mean what it visually appears to mean. When a literal hash octet belongs in the URI pattern, use its percent-encoded form where appropriate.
Fragments in normal web URLs are not sent to the server in HTTP requests, so they are usually not something robots.txt can control as separate resources anyway.
* and $ Are Matching Operators, Not Ordinary Path Characters
RFC 9309 defines * as matching zero or more characters and $ as marking the end of a match pattern. If the actual resource path contains a literal asterisk or dollar sign, percent-encode that octet rather than expecting it to be matched literally.
A Sitemap Line Helps Discovery; It Does Not Change the Crawl Rules Above It
Sitemap records are widely supported additional robots.txt records. RFC 9309 explicitly allows crawlers to interpret other records and says those records must not interfere with parsing standard groups and rules.
Listing a sitemap therefore does not “allow” URLs that a matching Disallow rule blocks. Discovery and crawler access are separate concerns.
/robots.txt location.