Meta Robots Tag: Controlling Indexing and Link-Following Per Page
The meta robots tag (<meta name="robots" content="...">) is a page-level directive controlling whether a crawler may index a page and whether it should follow the outbound links on it. It's independent of robots.txt, which governs crawling access rather than indexing.
Why it matters
A stray noindex from staging, or a mismatch between robots.txt and the meta tag, silently wrecks entire site sections. Because this tag often lives at the template level, one bad default can deindex hundreds of pages without anyone noticing until traffic tanks.
Google checks this tag before indexing; a page can pass crawl inspection but still get blocked by noindex. It's independent of robots.txt too. A page allowed by your robots file can still be nixed by a meta directive.
How it works
Place a <meta name="robots" content="..."> tag in the head. The content attribute accepts space-separated directives. The main directives are index (allow indexing), noindex (block indexing), follow (crawl outbound links), and nofollow (do not crawl outbound links).
Google reads this tag before indexing a page. If noindex is present, Google will not add the page to its index, but Google may still crawl it. You can also specify per-bot directives using the tag name (e.g., <meta name="googlebot" content="noindex">). The X-Robots-Tag HTTP response header serves the same function and takes precedence if both the meta tag and header are present.
What does not matter
noindex doesn't stop crawling; Google still crawls the page to check the directive. Don't use robots.txt to block indexing: use this tag instead. Only block crawling in robots.txt if you're trying to save server resources.
nofollow doesn't prevent Google from discovering a page through other links. It only says "don't follow this specific link." The page can still rank if linked elsewhere without nofollow.
Code example
Good: Block a staging or internal-only page from indexing.
<head>
<meta name="robots" content="noindex, nofollow">
</head>
Broken: A forgotten noindex directive left on a live page.
<head>
<meta name="robots" content="noindex">
<!-- Page is live but will never appear in search results -->
</head>
Auditing your robots directives
Start by checking your templates and staging environment. A good habit: audit any page you update and explicitly verify you're not inheriting a noindex from an old default. Check both the meta tag and the X-Robots-Tag HTTP header; if both are present, the header wins. If you've ever deindexed a staging site and forgot to flip the tag back, you know this lesson the hard way.