Skip to content

thedaviddias/Front-End-Checklist / skills/robots-meta-conflict

robots-meta-conflict

Verified

Use when applies to sites that use both robots.txt disallow rules and meta robots noindex tags. Use when investigating why deindexed pages still appear in search results.

FileKindSize
SKILL.mdskill md2.2 KB
references/rule.mdmarkdown3.6 KB

references/rule.md

View on GitHub
# Robots Meta Conflict

> Detects pages blocked by robots.txt that also carry noindex meta tags, creating a paradox where the directive is never read.

**Priority:** high · **Difficulty:** intermediate · **Time:** 10 min

---
A robots.txt `Disallow` rule and a `noindex` meta tag serve different purposes. [Google's robots-meta documentation](https://developers.google.com/search/docs/crawling-indexing/robots-meta-tag) makes that distinction explicit, which is why this conflict often needs the same cleanup as broader [indexability-conflicts](/en/rules/seo/indexability-conflicts).

## Code Example

```
# robots.txt
User-agent: *
Disallow: /private/

# /private/page.html
<meta name="robots" content="noindex">   ← Google never reads this
```

When Googlebot sees `Disallow: /private/`, it stops crawling that path entirely. It never fetches `/private/page.html`, so it never processes the `noindex` tag.

Result: Google **may still index the URL** using signals from external backlinks, sitelinks, or previously cached content.

## Why It Matters

Pages that appear in search results despite your intention to deindex them are usually blocked in robots.txt. Google cannot read the noindex directive it cannot fetch, so [Google Search Console](https://search.google.com/search-console/about) becomes the fastest way to confirm what Google actually saw.

## Correct Patterns

#

## ✅ Deindex a page (make it crawlable but excluded from results)

```
# robots.txt — NO Disallow for this path
User-agent: *
Disallow: /admin/
```

```html
<!-- /thank-you.html -->
<head>
  <meta name="robots" content="noindex, follow">
</head>
```

Googlebot fetches the page, reads `noindex`, and removes it from search results.

### ✅ Block a staging environment completely

```
# robots.txt on staging.example.com
User-agent: *
Disallow: /
```

This is acceptable on staging/preview environments where you simply do not want any content crawled at all.

### ❌ Conflicting setup (noindex never processed)

```
# robots.txt
User-agent: *
Disallow: /old-page/
```

```html
<!-- /old-page/index.html -->
<meta name="robots" content="noindex">   ← Never read by Googlebot
```

## Quick Reference

| Goal | robots.txt | meta robots |
|------|-----------|-------------|
| Allow crawling and indexing | No Disallow | `index, follow` (default) |
| Deindex page, allow link equity | No Disallow | `noindex, follow` |
| Block crawling (no deindex guarantee) | `Disallow: /path/` | (irrelevant — not read) |
| Block private tool content | `Disallow: /path/` | (not needed) |

## Exceptions

- Staging, utility, login, account, or internal search pages may intentionally use different crawl or index signals if they are not meant to rank.
- Temporary migration states can produce noisy intermediate signals; flag the live production URL pattern, not one-off transition artifacts.
- When redirects, canonicals, robots directives, or indexability signals conflict, fix the strongest final signal first instead of reporting every downstream symptom as a separate blocker.

## Standards

- Use these references as the standard for the final search-facing HTML, metadata, and crawl behavior.
- Check the implementation against Google: Remove a page from Google before treating the rule as satisfied.
- Check the implementation against Google: robots meta tag documentation before treating the rule as satisfied.

## Verification

### Automated Checks

- Check Google Search Console → URL Inspection Tool for affected URLs
- Use the robots.txt tester to verify crawlability

### Manual Checks

- Fetch the page as Google to confirm the meta robots tag is present and readable
slack-gif-creatoranthropics/skillsKnowledge and utilities for creating animated GIFs optimized for Slack. Provides constraints, validation tools, and animation concepts. Use when users request animated GIFs for Slack like "make me a GIF of X doing Y for Slack."Stars: 198kUpdated Claude CodeCodexVerifiedInstall
ponytailDietrichGebert/ponytailLazy senior dev mode for any coding task: the smallest change that fully solves it, and a reply a busy human understands in one read.Stars: 194kUpdated Claude CodeCodexVerifiedInstall
ponytail-debtDietrichGebert/ponytailHarvest every shortcut: comment into one debt ledger, so deferrals get tracked instead of forgotten. One-shot report.Stars: 194kUpdated Claude CodeCodexVerifiedInstall
ponytail-debtDietrichGebert/ponytailList every `shortcut:` comment (and older `ponytail:` ones) as a debt ledger. One-shot report, changes nothing. Use for "ponytail debt", "what did ponytail defer", "list the shortcuts", /ponytail-debt.Stars: 194kUpdated Claude CodeCodexVerifiedInstall