X-Robots-Tag

## Why it matters

X-Robots-Tag is the right tool for PDFs and images, but also a classic silent killer. A staging rule that leaks to production, or an edge rule nobody remembers, can noindex a whole path with nothing visible in the page. It only appears if you check headers, which HTML-only audits never do.

A staging server accidentally serving X-Robots-Tag: noindex in production means your pages vanish from Google silently. The browser shows them fine. Google just won't index them. Header-level auditing is the only way to catch this.

## How it works

X-Robots-Tag is an HTTP header with the same directive vocabulary as the robots meta tag. You can scope it per bot: X-Robots-Tag: googlebot: noindex applies only to Googlebot. X-Robots-Tag: noindex applies to all bots. Set multiple directives in one header or use multiple headers.

When both header and meta tag exist, the most restrictive rule wins. Header says noindex, tag says index? Noindexed. The header is server-controlled (authoritative), the tag is template-controlled. Server always wins.

The X-Robots-Tag header on the final hop of a redirect chain determines indexability. A noindex on an intermediate hop is meaningless (the page never indexes anyway), but it signals config sprawl. Intermediate responses should pass through cleanly without touching indexability rules.

## What does not matter

X-Robots-Tag is not a security tool. It doesn't prevent anyone from reading a page; it asks search engines not to index it. Crawlers can ignore it; users can always read the page in their browser. Noindex is a signal, not a firewall.

Robots.txt interaction: if robots.txt blocks a URL, search engines never see its X-Robots-Tag header. A noindex header on a blocked URL is pointless (though harmless). A URL appears unindexed, but the cause might be robots.txt, not the noindex header.

## Code example

A basic X-Robots-Tag header on all responses:

add_header X-Robots-Tag "noindex" always;
Copy

Applying noindex only to Google:

add_header X-Robots-Tag "googlebot: noindex" always;
Copy

Blocking a specific path (e.g., PDFs in /downloads/) from indexing while allowing other robots to cache it:

location ~* ^/downloads/ {
  add_header X-Robots-Tag "noindex";
  add_header Cache-Control "public, max-age=3600";
}
Copy

Conflicting headers: when a page has both meta and header directives:

<meta name="robots" content="index, follow" />
Copy

With this header set on the response:

X-Robots-Tag: noindex
Copy

Result: the page is noindexed. The header's noindex overrides the meta tag's index directive.

To check X-Robots-Tag on a page:

curl -sI https://example.com/path | grep X-Robots-Tag
# X-Robots-Tag: noindex
Copy

## How Scalpel shows it

Scalpel displays an X-Robots-Tag hint on any hop that carries the header, including intermediate hops HTML-only audits miss. You can spot a noindex on a mid-chain redirect (usually config sprawl) and verify the final hop has the correct directive, or none if the page should be indexed.