thedaviddias/Front-End-Checklist / skills/robots-meta-conflict
robots-meta-conflict
VerifiedUse when applies to sites that use both robots.txt disallow rules and meta robots noindex tags. Use when investigating why deindexed pages still appear in search results.
| File | Kind | Size |
|---|---|---|
| SKILL.md | skill md | 2.2 KB |
| references/rule.md | markdown | 3.6 KB |
references/rule.md
View on GitHub# Robots Meta Conflict
> Detects pages blocked by robots.txt that also carry noindex meta tags, creating a paradox where the directive is never read.
**Priority:** high · **Difficulty:** intermediate · **Time:** 10 min
---
A robots.txt `Disallow` rule and a `noindex` meta tag serve different purposes. [Google's robots-meta documentation](https://developers.google.com/search/docs/crawling-indexing/robots-meta-tag) makes that distinction explicit, which is why this conflict often needs the same cleanup as broader [indexability-conflicts](/en/rules/seo/indexability-conflicts).
## Code Example
```
# robots.txt
User-agent: *
Disallow: /private/
# /private/page.html
<meta name="robots" content="noindex"> ← Google never reads this
```
When Googlebot sees `Disallow: /private/`, it stops crawling that path entirely. It never fetches `/private/page.html`, so it never processes the `noindex` tag.
Result: Google **may still index the URL** using signals from external backlinks, sitelinks, or previously cached content.
## Why It Matters
Pages that appear in search results despite your intention to deindex them are usually blocked in robots.txt. Google cannot read the noindex directive it cannot fetch, so [Google Search Console](https://search.google.com/search-console/about) becomes the fastest way to confirm what Google actually saw.
## Correct Patterns
#
## ✅ Deindex a page (make it crawlable but excluded from results)
```
# robots.txt — NO Disallow for this path
User-agent: *
Disallow: /admin/
```
```html
<!-- /thank-you.html -->
<head>
<meta name="robots" content="noindex, follow">
</head>
```
Googlebot fetches the page, reads `noindex`, and removes it from search results.
### ✅ Block a staging environment completely
```
# robots.txt on staging.example.com
User-agent: *
Disallow: /
```
This is acceptable on staging/preview environments where you simply do not want any content crawled at all.
### ❌ Conflicting setup (noindex never processed)
```
# robots.txt
User-agent: *
Disallow: /old-page/
```
```html
<!-- /old-page/index.html -->
<meta name="robots" content="noindex"> ← Never read by Googlebot
```
## Quick Reference
| Goal | robots.txt | meta robots |
|------|-----------|-------------|
| Allow crawling and indexing | No Disallow | `index, follow` (default) |
| Deindex page, allow link equity | No Disallow | `noindex, follow` |
| Block crawling (no deindex guarantee) | `Disallow: /path/` | (irrelevant — not read) |
| Block private tool content | `Disallow: /path/` | (not needed) |
## Exceptions
- Staging, utility, login, account, or internal search pages may intentionally use different crawl or index signals if they are not meant to rank.
- Temporary migration states can produce noisy intermediate signals; flag the live production URL pattern, not one-off transition artifacts.
- When redirects, canonicals, robots directives, or indexability signals conflict, fix the strongest final signal first instead of reporting every downstream symptom as a separate blocker.
## Standards
- Use these references as the standard for the final search-facing HTML, metadata, and crawl behavior.
- Check the implementation against Google: Remove a page from Google before treating the rule as satisfied.
- Check the implementation against Google: robots meta tag documentation before treating the rule as satisfied.
## Verification
### Automated Checks
- Check Google Search Console → URL Inspection Tool for affected URLs
- Use the robots.txt tester to verify crawlability
### Manual Checks
- Fetch the page as Google to confirm the meta robots tag is present and readableRelated skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| slack-gif-creatoranthropics/skillsKnowledge and utilities for creating animated GIFs optimized for Slack. Provides constraints, validation tools, and animation concepts. Use when users request animated GIFs for Slack like "make me a GIF of X doing Y for Slack." | Stars: 198k | Updated | Claude CodeCodex | Verified | Install |
| ponytailDietrichGebert/ponytailLazy senior dev mode for any coding task: the smallest change that fully solves it, and a reply a busy human understands in one read. | Stars: 194k | Updated | Claude CodeCodex | Verified | Install |
| ponytail-debtDietrichGebert/ponytailHarvest every shortcut: comment into one debt ledger, so deferrals get tracked instead of forgotten. One-shot report. | Stars: 194k | Updated | Claude CodeCodex | Verified | Install |
| ponytail-debtDietrichGebert/ponytailList every `shortcut:` comment (and older `ponytail:` ones) as a debt ledger. One-shot report, changes nothing. Use for "ponytail debt", "what did ponytail defer", "list the shortcuts", /ponytail-debt. | Stars: 194k | Updated | Claude CodeCodex | Verified | Install |