A robots.txt tester says Allowed. The page opens normally in your browser. It is even listed in your sitemap. Yet Google Search Console still says the URL is not indexed.
There is no contradiction: Allowed means the tested crawler may request that path under the robots.txt rules. It does not mean Google has indexed the page. A robots.txt check answers whether a crawler is permitted to request a path. It cannot tell you whether Google discovered that path, fetched it successfully, found an indexing restriction, chose another page as canonical, or decided to index it.
If the crawler is allowed, leave robots.txt alone for now. Inspect that exact URL in Search Console to see whether the next issue is discovery, fetching, a noindex instruction, a different selected canonical or a decision not to index. Then follow the evidence for that particular URL.
“Allowed” only confirms crawl permission
Suppose you’re testing this URL:
https://example.com/guides/api-auth
Your robots.txt file contains:
User-agent: *
Allow: /
A tester correctly reports Allowed. That means the applicable rules do not prohibit the selected crawler from requesting that path. It doesn’t mean the page was crawled yesterday—or ever. It doesn’t inspect the page’s HTTP response, metadata or content.
One detail is easy to miss: a test for the wildcard user-agent isn’t automatically a test of every named crawler. Specific user-agent groups can take precedence over the wildcard group. Check the Googlebot group, actual URL path and applicable matching rule before accepting the result. A locally pasted robots.txt file also isn’t proof that the same file is live at the site root.
For example:
User-agent: *
Disallow: /private/
User-agent: Googlebot
Allow: /
Googlebot’s applicable group can produce a different answer from the wildcard group. The Yoryantra Robots.txt Tester is useful for checking rules, matching paths and crawler groups. It cannot observe Google’s actual crawl history or indexing decision. That’s the boundary between a useful diagnostic tool and a misleading promise.
Open URL Inspection before editing anything else
For one important page, go to Google Search Console and inspect its exact URL. Don’t rely only on the site-wide Page indexing chart: it groups URLs by issue, while URL Inspection shows evidence about the URL you care about.
Look at the Page indexing result, Last crawl, Crawled as, the reported canonical URLs and the reason for exclusion. If you recently changed the page, compare the recorded crawl with Test Live URL. The live test can check current access and indexing eligibility, but it doesn’t mean Google has already indexed that version, nor does a successful result guarantee inclusion.
It’s also worth writing down when the page was published or substantially changed. A new page that hasn’t been crawled yet calls for different action from a previously crawled page whose content Google declined to index.
If Google hasn’t discovered the URL
Google needs to find a URL before it can crawl it. Crawl permission doesn’t create a discovery path.
Imagine publishing /guides/api-auth but leaving it outside your site navigation, related-article links and sitemap. You know the URL; Google may not. Check whether another crawlable page links to it using a normal link, and whether the URL appears in the intended XML sitemap.
A sitemap is helpful, especially for large or recently updated sites, but submission is a hint, not a command. A “Success” status for a sitemap means Google could process it; it does not mean every listed URL was fetched or indexed. If the URL is linked properly and submitted, allow time for discovery instead of repeatedly altering robots.txt.
If Search Console reports Discovered – currently not indexed, Google knows about the URL but has not crawled it yet. That is different from a page that was crawled and not selected for the index. Investigate discovery paths, site availability and possible crawl scheduling issues; don’t assume a missing Allow rule is the problem.
If Google cannot fetch the page
An Allowed decision means a request is permitted by the crawl rules. Your hosting setup still has to serve a usable response.
A page may work for you and fail for Googlebot because of an intermittent 5xx, DNS failure, timeout, authentication requirement, CDN security challenge, firewall rule or redirect chain. A 401 or 403 response is an access problem, even if robots.txt allows the URL. And a 200 response is worth checking for the content Google actually receives, not just its status code.
A practical check is to compare three views: what your browser loads, what the server returns, and what Search Console’s live test reports. If the live test fails, investigate the error it shows. Don’t diagnose a hosting or security-layer failure by endlessly editing a crawl-rule file.
If the page tells Google not to index it
Two directives commonly explain an otherwise accessible page being excluded:
<meta name="robots" content="noindex">
X-Robots-Tag: noindex
The first is in the HTML; the second is an HTTP response header. For a page intended to appear in Search, an unintended noindex in either place is a real blocker. Examine both the delivered HTML and response headers, including whether a deployment environment or middleware added an instruction you didn’t expect.
Here’s the important distinction: robots.txt is a crawl control, while noindex is an indexing instruction Google has to discover when it accesses the resource. Blocking a page in robots.txt can stop Google from seeing its noindex. That means combining Disallow with noindex is not a reliable way to remove a normal web page from search results.
If removal is your goal, follow Google’s removal and noindex guidance. If inclusion is your goal, remove accidental indexing restrictions and keep the resource crawlable. Yoryantra’s Indexability Checker can help examine pasted HTML and response headers, but it cannot replace Search Console’s observation of the live URL.
If Google has chosen another canonical URL
A technically sound page can remain unindexed as a separate URL because Google regards another page as the representative version.
Consider:
https://example.com/guides/api-auth
https://example.com/guides/api-auth?source=nav
Both may return 200, permit crawling and contain no noindex. If their main content is effectively the same, Google may group them and index only the selected canonical. That’s usually appropriate deduplication—not necessarily an indexing defect.
In URL Inspection, compare the user-declared canonical with the Google-selected canonical. Then check your redirects, internal links, sitemap entries and rel="canonical" elements. A self-referencing canonical is useful, but it is a signal, not a guarantee that Google will make the same choice. The Yoryantra Canonical URL Checker can help with the canonical declaration; Search Console is where you check Google’s actual selection.
Don’t try to solve two near-identical pages by giving each one a stronger robots.txt permission. Both were already allowed. Decide whether they should really be separate pages or whether one canonical version is sufficient.
Read the exclusion reason, not just “Not indexed”
Search Console’s labels help narrow down what to do next. They are not all interchangeable.
| Search Console finding | What it tells you | Next useful check |
|---|---|---|
| Blocked by robots.txt | The applicable rules prevent crawling | Inspect the winning rule and relevant crawler group |
| Excluded by noindex | Google found an instruction not to index | Check HTML meta robots and X-Robots-Tag |
| Discovered – currently not indexed | Google knows the URL but has not crawled it | Check internal links, sitemap, availability and timing |
| Crawled – currently not indexed | Google crawled the page but has not indexed it | Review content, duplication, canonical choice and page interpretation |
| Alternate/duplicate canonical | Google prefers another representative URL | Compare declared and Google-selected canonicals |
| Soft 404 | Google interprets the page as effectively missing or unhelpful for that URL | Check rendered content and proper HTTP status |
| Server or access error | Google couldn’t retrieve a usable page | Inspect CDN, authorization, infrastructure and redirects |
Crawled – currently not indexed deserves special care. It does not have a single guaranteed cause. Google may index the page later, and Google’s documentation says you don’t need to resubmit that URL for crawling simply because it carries this status. If it matters to your site, compare the content with similar indexed pages, check the canonical decision and look for a thin, duplicative or misleading page experience. Don’t turn a report label into an invented diagnosis.
A soft 404 is also different from a network failure. The server can return 200 while the page looks like a missing result—perhaps a generic “nothing here” template. Serving the correct meaningful content, or an appropriate error status for a genuinely missing page, is more useful than changing robots.txt.
Why a blocked URL can sometimes still appear in Search
There’s a reverse case that surprises people: a URL is disallowed in robots.txt but still appears in search results.
Google can learn that a URL exists from links and other signals even when it cannot crawl the page itself. A block can prevent Google from seeing the content without necessarily preventing the URL from appearing in results. This is why Disallow is not a dependable “remove from Google” control, and why a crawler must be able to access a noindex directive for that directive to work.
The two facts fit together:
-
An Allowed URL isn’t guaranteed to be indexed.
-
A Disallowed URL isn’t guaranteed to stay absent from search results.
Allowing Googlebot to crawl a URL does not guarantee that Google will index it.
Check one missing URL from discovery to indexing
When you have an article or tool page that should be indexed, use this sequence and stop as soon as you find a real blocker.
-
Confirm the exact URL and crawler. Test the currently deployed robots.txt rules for the right path and user-agent. If allowed, move on.
-
Inspect the URL in Search Console. Read its recorded indexing reason, last crawl and canonical details. Distinguish historical crawl information from a live test.
-
Verify Google’s access. Check redirects, final HTTP status, security challenges and the content that reaches Googlebot.
-
Inspect indexing directives. Look for unintended noindex in HTML and response headers. Confirm that the page renders its useful main content.
-
Check discovery and canonical consistency. Ensure sensible internal links, sitemap inclusion where appropriate, and a consistent preferred URL. Compare with closely similar pages.
-
Act on the actual finding. Fix a bad robots rule, access restriction, wrong directive, duplicate or ineffective page. If everything is technically sound, don’t manufacture a robots.txt problem to explain Google’s indexing choice.
After a meaningful fix, the URL Inspection Request Indexing feature may be appropriate for an important page. It is not a force-index button; repeated requests don’t make crawling faster. For the Crawled – currently not indexed status in particular, resubmission without a change usually adds no insight. Monitor subsequent crawls and decisions rather than treating the request confirmation as proof of indexing.
What the tester contributes—and what it can’t
The Yoryantra Robots.txt Tester answers a precise question: given a set of pasted rules, a selected crawler and a URL path, which crawl rule applies? That is valuable when a wrong group or wildcard match is blocking a useful page.
For the rest of the diagnosis, use the appropriate evidence: URL Inspection for Google’s recorded view, live fetch tests for current accessibility, an indexability check for page directives, and a canonical check when duplicate URLs are involved. These tools complement one another because they observe different layers.
If you remember just one distinction, make it this: Allowed means Googlebot may request the URL. Indexed means Google has chosen to include a canonical page in its index. The space between those outcomes is where you should investigate.