thedaviddias/Front-End-Checklist / skills/robots-meta-conflict
robots-meta-conflict
VerifiedUse when applies to sites that use both robots.txt disallow rules and meta robots noindex tags. Use when investigating why deindexed pages still appear in search results.
| File | Kind | Size |
|---|---|---|
| SKILL.md | skill md | 2.2 KB |
| references/rule.md | markdown | 3.6 KB |
SKILL.md
View on GitHub---
name: robots-meta-conflict
description: "Use when applies to sites that use both robots.txt disallow rules and meta robots noindex tags. Use when investigating why deindexed pages still appear in search results."
metadata:
category: seo
priority: high
difficulty: intermediate
estimatedTime: "10"
source: frontendchecklist.io
url: https://frontendchecklist.io/rules/seo/robots-meta-conflict
---
# Robots Meta Conflict
Pages that appear in search results despite your intention to deindex them are usually blocked in robots.txt — Google cannot read the noindex directive it cannot fetch.
## Quick Reference
- If a page is blocked in robots.txt, crawlers never fetch it and therefore never read its `noindex` meta tag
- To properly deindex a page, use `<meta name="robots" content="noindex">` without blocking it in robots.txt
- If you block a URL in robots.txt, Google may still show it in results using anchor text from backlinks
- Audit pages that need to be deindexed by confirming they are crawlable but carry `noindex`
## Check
Cross-reference the site's robots.txt disallow paths against pages carrying `<meta name="robots" content="noindex">`. Flag any page where the URL matches a Disallow rule AND has a noindex tag — the noindex will never be seen by crawlers.
## Fix
For pages that must be deindexed: remove the robots.txt Disallow rule for that URL and keep the `noindex` meta tag. For pages that must simply be uncrawled without appearing: keep the Disallow rule and remove the noindex tag (understanding Google may still show the URL).
## Explain
Explain why a page blocked by robots.txt can still appear in Google search results, why the noindex meta tag inside a blocked page is never processed, and how to correctly deindex a page.
## Code Review
Review metadata generation, rendered HTML, structured data, and response headers related to Robots Meta Conflict. Flag exact routes or templates where search-facing output violates the rule, and describe how to verify the final page output.
---
For full implementation details, code examples, and framework-specific guidance,
see `references/rule.md`.
Rule page: https://frontendchecklist.io/rules/seo/robots-meta-conflict
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| slack-gif-creatoranthropics/skillsKnowledge and utilities for creating animated GIFs optimized for Slack. Provides constraints, validation tools, and animation concepts. Use when users request animated GIFs for Slack like "make me a GIF of X doing Y for Slack." | Stars: 198k | Updated | Claude CodeCodex | Verified | Install |
| ponytailDietrichGebert/ponytailLazy senior dev mode for any coding task: the smallest change that fully solves it, and a reply a busy human understands in one read. | Stars: 194k | Updated | Claude CodeCodex | Verified | Install |
| ponytail-debtDietrichGebert/ponytailHarvest every shortcut: comment into one debt ledger, so deferrals get tracked instead of forgotten. One-shot report. | Stars: 194k | Updated | Claude CodeCodex | Verified | Install |
| ponytail-debtDietrichGebert/ponytailList every `shortcut:` comment (and older `ponytail:` ones) as a debt ledger. One-shot report, changes nothing. Use for "ponytail debt", "what did ponytail defer", "list the shortcuts", /ponytail-debt. | Stars: 194k | Updated | Claude CodeCodex | Verified | Install |