Skip to main content
Yoryantra
Home/Tools
← Back to Tools

Meta Robots Tag Generator

Build robots meta and X-Robots-Tag directives for indexing, link following, snippets, previews, and expiry.

Page Indexing Settings

A crawler must be able to fetch the URL to discover noindex. For confidential or staging content, authentication or network access control is still required; robots directives are not a security layer.

Output

Generated meta robots output will appear here.
Directive generation runs in the browser. The selected settings are not sent to Yoryantra for processing.

Decide whether you are controlling indexing, previews, or both

noindex controls whether a URL may appear in supported search indexes. Preview directives such asnosnippet,max-snippet, andmax-image-preview control how much content a crawler may show when the URL is otherwise eligible.

Explicit index, follow values are normally unnecessary because they are defaults for Google and Bing. They remain available here when an explicit configuration is easier to audit alongside other directives.

noindex does not make confidential content private

Search directives are not authentication. A crawler must also be able to fetch a URL to discover noindex, so blocking that same URL in robots.txt can prevent the directive from being seen. Protect private dashboards, staging sites, and sensitive files with access control.

HTML pages can use a robots meta element

Place the robots meta element in a valid HTML head. A crawler-specific name such as googlebot or bingbot is only needed when that crawler should receive rules different from the generic robots policy.

Non-HTML resources need X-Robots-Tag

PDFs, images, and other non-HTML responses cannot carry an HTML meta element. Send the equivalent rule as an HTTP X-Robots-Tag header and verify the deployed response rather than only the application code.

Preview limits have special values

  • max-snippet:0 suppresses a text snippet; -1 means no limit.
  • max-video-preview:0 prevents a video preview; -1 allows any available length.
  • max-image-preview accepts none, standard, or large.
  • unavailable_after needs a valid date/time; ISO 8601 is a clear choice for deployment.

noarchive now means different things across search engines

Google Search documents noarchive as an ignored historical rule after removing its cached-link feature. Bing still documents noarchive and nocache behavior. Keep it only when the crawler you care about still supports it, and verify current documentation before relying on it.

Duplicate URLs usually need a canonical decision first

A duplicate URL is not automatically a noindex candidate. If the goal is to consolidate equivalent pages while keeping links and discovery intact, review canonicalization and redirects before removing the URL from search entirely.

Framework output preserves the full directive string

Next.js accepts the Metadata robots field as a string, so the generated snippet keeps the selected generic directives together instead of silently dropping preview or expiry settings. Identical crawler-specific tags are redundant when the generic robots value already applies. Search engines do not all support every directive, so check the crawler-specific documentation when a rule is business-critical.