How to find and remove a noindex tag or robots.txt block that keeps your site out of Google

Your site is not missing from Google. It is being refused, on your own instruction. Three mechanisms can issue it, and two leave nothing in your HTML.

Your site is not missing from Google. It is being refused, by your own server, on your own instruction. Three separate mechanisms can issue that instruction, they live in three different places, and two of them leave nothing in your HTML to find. Removing the block takes minutes. Locating it is the work.

What this actually is

Crawling and indexing are two different permissions, and people treat them as one. That confusion is the whole problem.

robots.txt controls the request. It tells a crawler which URLs it may fetch. noindex controls what happens after the fetch: Google may read the page, and may not list it.

So the two rules do opposite things, and combining them backfires. A URL under Disallow is never fetched, so the noindex inside it is never read. Google's own documentation is direct about this: for the noindex rule to work, the page must not be blocked by robots.txt and must otherwise be reachable by the crawler.

Worse, a disallowed URL can still show up in search results. Google will not request the page, but it can still index the URL using information from the pages that link to it. Search Console has a label for exactly this outcome, and Google's own note on it is that the snippet shown for such a page will probably be very limited. You get a listing with almost nothing in it, for a page you thought you had hidden.

noindex itself arrives one of two ways, and they are equally binding:

- A <meta name="robots" content="noindex"> tag in the <head> of the HTML.
- An X-Robots-Tag: noindex HTTP response header, which never appears in view-source.

The header is the one that costs an afternoon. It can be set by the application, by the web server config, by a reverse proxy, or by the content delivery network (CDN) edge, and the first three places you look will not be the right one.

How to check

Four checks. Run all four, on the exact hostname and scheme people actually visit.

1. The response header. From any terminal:

curl -sI -A "Mozilla/5.0" https://example.com/ | grep -i "x-robots-tag"

Silence means no header. Any output is your answer. Add -L to follow redirects, and repeat against a deep page: some setups apply the header per-route.

2. The HTML.

curl -s -A "Mozilla/5.0" https://example.com/ | grep -i "noindex"

This reads the raw HTML, which is what Googlebot fetches first. If your pages are rendered by JavaScript, also check the rendered DOM in Chrome DevTools (Elements panel, search for noindex) - and know that a JavaScript fix will not save you. Google states that when it encounters noindex, it may skip rendering and JavaScript execution entirely, so removing the tag with a script may not work.

3. robots.txt.

curl -s -A "Mozilla/5.0" https://example.com/robots.txt

The file is only valid for the exact host, protocol, and port it was fetched from. https://example.com/robots.txt says nothing about https://www.example.com/ or http://example.com/. Check every variant that resolves.

4. The browser network tab. Open DevTools, Network, reload the page, click the first document request, and read Response Headers. This is the same data as curl -I, and it catches the case where a CDN treats a browser differently from a command-line client.

Then confirm against Google's own view: Search Console, URL Inspection, and read Coverage > Indexing > Indexing allowed?. Click Test live URL to see the current state rather than the last-crawl state; on the live result the same field sits under Availability > Indexing.

Site-wide, two reports cover two different questions. Page indexing gives you URL-level reasons, and it is where "URL blocked by robots.txt" and "URL marked 'noindex'" appear. Settings > robots.txt tells you only whether Google could fetch and parse your robots.txt files: it lists the top 20 hosts by crawl rate, and it exists only on domain properties. A URL-prefix property checks one origin and nothing else.

Reading robots.txt without guessing

User-agent opens a group. Disallow and Allow take a path relative to the site root. Disallow: / blocks everything; Disallow: with an empty value blocks nothing.

Only one user agent wins: the crawler follows the group with the most specific matching user agent, and non-matching groups are ignored. A User-agent: * group is not combined with a User-agent: Googlebot group, so the moment you add a Googlebot group, the global one stops applying to Googlebot. Rules do stack in one case: if the same user agent is declared in more than one group, Google merges those groups internally and applies all of their rules together.

Two wildcards are supported: * matches any sequence of characters, and $ anchors the end of the URL. So Disallow: /*.htm blocks every path containing .htm, and Allow: /$ allows only the root.

Precedence is not top-to-bottom. Google uses the most specific rule, measured by the length of the rule path, and on a genuine tie it uses the least restrictive one. Google's documented examples:

- allow: /p beats disallow: / for /page: more specific.
- allow: /folder beats disallow: /folder for /folder/page: tie, so least restrictive.
- disallow: /*.htm beats allow: /page for /page.htm: longer path match.
- disallow: / beats allow: /$ for /page.htm: the $ rule only covers the root.

One rule that no longer exists: noindex: inside robots.txt. Google retired support for it on September 1, 2019. If your file has that line, it is doing nothing, and the block is somewhere else.

What you will see

robots.txt returns 404. No crawl restrictions. Not your problem. Note that a 5xx on robots.txt is a different animal, and it runs on a schedule. For the first 12 hours Google stops crawling the site while it keeps retrying the file. For the next 30 days it uses the last good version it has cached, and if it has no cached version it assumes there are no crawl restrictions. After 30 days it depends on the rest of the site: if the site is generally reachable Google behaves as if there is no robots.txt file, and if the site has general availability problems Google stops crawling it.

Disallow: / under User-agent: *. The whole site is uncrawlable.

X-Robots-Tag: noindex, or a meta robots tag. The page is crawlable and unlistable.

Search Console's Page indexing report uses these exact labels:

- URL blocked by robots.txt - the page was not fetched.
- URL marked 'noindex' - the page was fetched, and a noindex directive was found in the HTML or the response headers.
- Indexed, though blocked by robots.txt - the trap, live. Someone linked to the page, so Google indexed the URL without ever loading it. The snippet will be thin.
- Discovered - currently not indexed - not a block. Google found the URL and has not crawled it yet.
- Blocked due to unauthorized request (401) / Blocked due to access forbidden (403) - a fourth family of blocks: staging authentication that shipped with everything else.

URL Inspection adds what the report cannot: Crawl allowed? and Indexing allowed? side by side, with a stated reason when either is No. One trap sits in that pair. If robots.txt blocks the URL, Indexing allowed? reads Yes regardless of what is in the page, because Google never fetched it and never saw the noindex. Read Crawl allowed? first, and treat a Yes on indexing as meaningless until crawling is allowed. To identify which mechanism issued the noindex, Google's own instruction is to search the page source and the response headers for the word, which is what checks 1 and 2 do.

How to fix

Remove the block at its source. The order matters when both are present: remove the Disallow first, so the crawler can reach the page and see the current state, then remove the noindex.

Where the instruction is issued, by platform:

- WordPress: Settings > Reading > Search Engine Visibility. What the "Discourage search engines from indexing this site" checkbox does depends on the version. From WordPress 5.3 on it writes <meta name='robots' content='noindex,nofollow' /> into the head, and only if the theme calls wp_head. Through WordPress 5.2 it instead made robots.txt return User-agent: * and Disallow: /, and only when WordPress sits in the site root with no real robots.txt file present. So on an older install the page source can look clean while the whole site is disallowed. Uncheck the box either way. SEO plugins add a second, independent per-page toggle, so check both.
- Next.js: the robots field in generateMetadata or the exported metadata object, usually gated on an environment variable that is set wrong in production.
- Apache: Header set X-Robots-Tag "noindex" in httpd.conf, a vhost, or any .htaccess up the directory tree. Grep the whole config tree, not just the file you remember editing.
- nginx: add_header X-Robots-Tag noindex; in server or location blocks. Inherited add_header directives are dropped when a nested block declares its own, which is how these survive a partial cleanup.
- Caddy: a header directive in the Caddyfile.
- CDN or host edge config: Netlify netlify.toml, Vercel headers, Cloudflare Transform Rules. This is the layer people forget, because the block is not in the application repository at all.

Then deploy, and verify with the same curl -I you started with. If the header persists after a code deploy, purge the CDN cache and try again.

Two more places to clear:

- Search Console Removals: an outstanding temporary removal request hides a URL for about six months, entirely independent of your HTML. Check the tool and cancel any request that is no longer wanted.
- Basic auth or IP allowlists left over from staging, which produce 401 and 403 rather than noindex.

Then request a recrawl: URL Inspection, then Request Indexing. Do the homepage and your three or four most important pages. There is a quota on individual submissions, so do not queue the whole site this way. Submit or resubmit your sitemap for the rest.

What it costs

Checking: 15 minutes for all four checks across every hostname variant.

Finding the source: 10 minutes if it is in the HTML. Half a day if it is a response header injected somewhere in the chain between your application and the browser, because you have to walk that chain layer by layer.

Fixing: one config change and one deploy.

Waiting: Google caches robots.txt for up to 24 hours, so a robots.txt fix is not live to Googlebot the moment you push it. After that, Google's own estimate for recrawl is a few days to a few weeks, and it publishes no tighter number and no priority you can buy. Plan on weeks and be pleased if it is days.

Verifying: do not assume. A week after the fix, open the Page indexing report and confirm the URLs have moved out of "URL blocked by robots.txt" and "URL marked 'noindex'". The absence of the tag on your laptop is not proof; the report is.

Not sure which of the four checks applies to your site? Run a free web audit and we will tell you exactly which of the three hiding places is holding your pages back.