thedaviddias/Front-End-Checklist / skills/indexability
indexability
VerifiedUse when auditing a site for accidental noindex directives, investigating why important pages are not appearing in Google Search, or reviewing CMS settings that control indexing at the page or category level.
| File | Kind | Size |
|---|---|---|
| SKILL.md | skill md | 3.2 KB |
| references/rule.md | markdown | 4.1 KB |
references/rule.md
View on GitHub# Make important pages indexable
> Identifies important pages blocked from search engine indexing by noindex, robots.txt, or other directives
**Priority:** medium · **Difficulty:** intermediate · **Time:** 10 min
---
For a page to appear in Google Search, it must be crawlable and indexable. [Google's indexing controls guide](https://developers.google.com/search/docs/crawling-indexing/block-indexing) and your broader [robots meta strategy](/en/rules/seo/robots-meta) both matter here, because three separate mechanisms can block indexing: `noindex` meta tags, `X-Robots-Tag` HTTP headers, and `robots.txt` rules.
## Code Examples
#
## ❌ Avoid — important page accidentally noindexed
```html
<!-- Product page that should rank in search -->
<meta name="robots" content="noindex, nofollow">
<!-- This page will never appear in Google Search -->
```
### ❌ Avoid — Next.js metadata with indexing disabled
```ts
// app/products/[slug]/page.tsx — accidental noindex
robots: {
index: false, // This blocks indexing for ALL product pages!
follow: false,
},
}
```
### ✅ Correct — indexable page (no noindex)
```html
<!-- Default: no meta robots tag = index, follow -->
<!-- Or be explicit: -->
<meta name="robots" content="index, follow">
```
### ✅ Correct — Next.js metadata enabling indexing
```ts
// For pages you want indexed, either omit 'robots' or set explicitly:
robots: {
index: true,
follow: true,
},
}
```
### ✅ Limiting noindex to genuinely private content
```ts
// pages/api/preview-post.ts — only noindex preview/draft pages
const post = await getPost(params.slug)
return {
robots: post.published
? { index: true, follow: true }
: { index: false, follow: false }, // Draft only
}
}
```
### ✅ Testing indexability
```bash
# Check X-Robots-Tag header
curl -I https://yoursite.com/important-page | grep -i 'x-robots'
# Check meta robots in HTML
curl -s https://yoursite.com/important-page | grep -i 'robots'
```
## Why It Matters
- **No index = no ranking**: A page with `noindex` cannot appear in search results under any circumstances.
- **Common accidents**: CMS platforms often default new pages to `noindex` (e.g., draft mode, staging environments accidentally promoted, category pages in WordPress).
- **Silent failure**: Indexing blocks cause no visible errors on the frontend. Only [Google Search Console](https://search.google.com/search-console/about) or log analysis reliably surfaces the problem, and those same checks help uncover [indexability conflicts](/en/rules/seo/indexability-conflicts).
## The Three Indexing Mechanisms
| Mechanism | Scope | Who reads it |
|---|---|---|
| `<meta name="robots" content="noindex">` | Single page | Crawlers that fetched the page |
| `X-Robots-Tag: noindex` HTTP header | Single resource | Crawlers that fetched the resource |
| `robots.txt Disallow` | URL pattern | Crawlers before fetching |
## Exceptions
- Staging, utility, login, account, or internal search pages may intentionally use different crawl or index signals if they are not meant to rank.
- Temporary migration states can produce noisy intermediate signals; flag the live production URL pattern, not one-off transition artifacts.
- When redirects, canonicals, robots directives, or indexability signals conflict, fix the strongest final signal first instead of reporting every downstream symptom as a separate blocker.
## Standards
- Use these references as the standard for the final search-facing HTML, metadata, and crawl behavior.
- Check the implementation against Google Search Central: Prevent Google from indexing pages before treating the rule as satisfied.
- Check the implementation against Google Search Central: Robots meta tag before treating the rule as satisfied.
## Verification
### Automated Checks
- Use Google Search Console Coverage report → "Excluded" tab to find pages marked "Indexed, though blocked by robots.txt" or "Noindex page".
- Use URL Inspection in Search Console for specific URLs.
### Manual Checks
- Export your top 50–100 pages by importance (traffic, revenue, links).
- For each, check: meta robots, X-Robots-Tag header, robots.txt.
- Fix blocks and request re-indexing.Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| rea-analysis-optimizationmorluto/reaOptimize REA providers and shared analysis workflows for agent context, latency, CPU, and memory while preserving correctness and Evidence. Use for analysis bottlenecks and cache or query design; CI/test-suite optimization belongs to rea-ci-tests-audit. | Stars: 472k | Updated | Claude CodeCodex | Verified | Install |
| rea-ci-tests-auditmorluto/reaAudit REA CI and test suites for correctness, fixture drift, false passes, and measured performance; use for suite or workflow investigations and efficiency changes, rather than ordinary feature testing or a single test failure. | Stars: 472k | Updated | Claude CodeCodex | Verified | Install |
| rea-tool-designmorluto/reaAudit, investigate, design, or change REA investigation tools, CLI/MCP contracts, provider capabilities, and Evidence semantics. | Stars: 472k | Updated | Claude CodeCodex | Verified | Install |
| ponytail-auditDietrichGebert/ponytailQuality audit of the whole repo: bugs, security, real load, missing tests, speed, and what to delete. Most important first. | Stars: 194k | Updated | Claude CodeCodex | Verified | Install |