Skip to content

thedaviddias/Front-End-Checklist / skills/all-noindex-pages

all-noindex-pages

Verified

Use when auditing metadata, crawlability, structured data, or indexability related to Audit all noindex pages. Verify the rendered HTML and HTTP response rather than relying only on source files.

FileKindSize
SKILL.mdskill md1.6 KB
references/rule.mdmarkdown2.3 KB

references/rule.md

View on GitHub
# Audit all noindex pages

> Lists and reviews all pages blocked from indexing to ensure critical content is accessible.

**Priority:** medium · **Difficulty:** intermediate · **Time:** 10 min

---
Regularly auditing your `noindex` directives ensures that you are only blocking pages that don't provide search value, such as staging environments, admin panels, or internal search results.

## Code Example

```html
<head>
  <!-- Use this for utility pages like 'Thank You' pages or internal search results -->
  <meta name="robots" content="noindex, follow">
  
  <!-- For pages that should be completely ignored -->
  <meta name="robots" content="noindex, nofollow">
</head>
```

## Why It Matters

- **Visibility**: Prevents accidental exclusion of high-value pages from Google and Bing.
- **Crawl Budget**: Helps search engines focus their limited crawl resources on your most important pages.
- **De-indexing**: Useful for removing outdated or sensitive pages from search results without deleting them.
- **Strategic Control**: Allows you to manage which versions of content (e.g., filtered lists) are available for indexing.

## Exceptions

- Staging, utility, login, account, or internal search pages may intentionally use different crawl or index signals if they are not meant to rank.
- Temporary migration states can produce noisy intermediate signals; flag the live production URL pattern, not one-off transition artifacts.
- When redirects, canonicals, robots directives, or indexability signals conflict, fix the strongest final signal first instead of reporting every downstream symptom as a separate blocker.

## Standards

- Use these references as the standard for the final search-facing HTML, metadata, and crawl behavior.
- Check the implementation against Google Search Central: Search Essentials before treating the rule as satisfied.
- Check the implementation against Google Search Central documentation before treating the rule as satisfied.

## Verification

### Automated Checks

- Inspect rendered HTML and HTTP headers to confirm the expected metadata or crawlability signal is present.
- Test the affected URL with Google Search Console or equivalent tooling where relevant.
- Re-crawl a representative page set after deployment.

### Manual Checks

- Confirm the change does not create conflicting canonical-url, robots, or structured-data signals.
rea-analysis-optimizationmorluto/reaOptimize REA providers and shared analysis workflows for agent context, latency, CPU, and memory while preserving correctness and Evidence. Use for analysis bottlenecks and cache or query design; CI/test-suite optimization belongs to rea-ci-tests-audit.Stars: 472kUpdated Claude CodeCodexVerifiedInstall
rea-ci-tests-auditmorluto/reaAudit REA CI and test suites for correctness, fixture drift, false passes, and measured performance; use for suite or workflow investigations and efficiency changes, rather than ordinary feature testing or a single test failure.Stars: 472kUpdated Claude CodeCodexVerifiedInstall
rea-tool-designmorluto/reaAudit, investigate, design, or change REA investigation tools, CLI/MCP contracts, provider capabilities, and Evidence semantics.Stars: 472kUpdated Claude CodeCodexVerifiedInstall
ponytail-auditDietrichGebert/ponytailQuality audit of the whole repo: bugs, security, real load, missing tests, speed, and what to delete. Most important first.Stars: 194kUpdated Claude CodeCodexVerifiedInstall