Skip to content

thedaviddias/Front-End-Checklist / skills/indexability

indexability

Verified

Use when auditing a site for accidental noindex directives, investigating why important pages are not appearing in Google Search, or reviewing CMS settings that control indexing at the page or category level.

FileKindSize
SKILL.mdskill md3.2 KB
references/rule.mdmarkdown4.1 KB

references/rule.md

View on GitHub
# Make important pages indexable

> Identifies important pages blocked from search engine indexing by noindex, robots.txt, or other directives

**Priority:** medium · **Difficulty:** intermediate · **Time:** 10 min

---
For a page to appear in Google Search, it must be crawlable and indexable. [Google's indexing controls guide](https://developers.google.com/search/docs/crawling-indexing/block-indexing) and your broader [robots meta strategy](/en/rules/seo/robots-meta) both matter here, because three separate mechanisms can block indexing: `noindex` meta tags, `X-Robots-Tag` HTTP headers, and `robots.txt` rules.

## Code Examples

#

## ❌ Avoid — important page accidentally noindexed

```html
<!-- Product page that should rank in search -->
<meta name="robots" content="noindex, nofollow">
<!-- This page will never appear in Google Search -->
```

### ❌ Avoid — Next.js metadata with indexing disabled

```ts
// app/products/[slug]/page.tsx — accidental noindex

  robots: {
    index: false,   // This blocks indexing for ALL product pages!
    follow: false,
  },
}
```

### ✅ Correct — indexable page (no noindex)

```html
<!-- Default: no meta robots tag = index, follow -->
<!-- Or be explicit: -->
<meta name="robots" content="index, follow">
```

### ✅ Correct — Next.js metadata enabling indexing

```ts

  // For pages you want indexed, either omit 'robots' or set explicitly:
  robots: {
    index: true,
    follow: true,
  },
}
```

### ✅ Limiting noindex to genuinely private content

```ts
// pages/api/preview-post.ts — only noindex preview/draft pages

  const post = await getPost(params.slug)

  return {
    robots: post.published
      ? { index: true, follow: true }
      : { index: false, follow: false },  // Draft only
  }
}
```

### ✅ Testing indexability

```bash
# Check X-Robots-Tag header
curl -I https://yoursite.com/important-page | grep -i 'x-robots'

# Check meta robots in HTML
curl -s https://yoursite.com/important-page | grep -i 'robots'
```

## Why It Matters

- **No index = no ranking**: A page with `noindex` cannot appear in search results under any circumstances.
- **Common accidents**: CMS platforms often default new pages to `noindex` (e.g., draft mode, staging environments accidentally promoted, category pages in WordPress).
- **Silent failure**: Indexing blocks cause no visible errors on the frontend. Only [Google Search Console](https://search.google.com/search-console/about) or log analysis reliably surfaces the problem, and those same checks help uncover [indexability conflicts](/en/rules/seo/indexability-conflicts).

## The Three Indexing Mechanisms

| Mechanism | Scope | Who reads it |
|---|---|---|
| `<meta name="robots" content="noindex">` | Single page | Crawlers that fetched the page |
| `X-Robots-Tag: noindex` HTTP header | Single resource | Crawlers that fetched the resource |
| `robots.txt Disallow` | URL pattern | Crawlers before fetching |

## Exceptions

- Staging, utility, login, account, or internal search pages may intentionally use different crawl or index signals if they are not meant to rank.
- Temporary migration states can produce noisy intermediate signals; flag the live production URL pattern, not one-off transition artifacts.
- When redirects, canonicals, robots directives, or indexability signals conflict, fix the strongest final signal first instead of reporting every downstream symptom as a separate blocker.

## Standards

- Use these references as the standard for the final search-facing HTML, metadata, and crawl behavior.
- Check the implementation against Google Search Central: Prevent Google from indexing pages before treating the rule as satisfied.
- Check the implementation against Google Search Central: Robots meta tag before treating the rule as satisfied.

## Verification

### Automated Checks

- Use Google Search Console Coverage report → "Excluded" tab to find pages marked "Indexed, though blocked by robots.txt" or "Noindex page".
- Use URL Inspection in Search Console for specific URLs.

### Manual Checks

- Export your top 50–100 pages by importance (traffic, revenue, links).
- For each, check: meta robots, X-Robots-Tag header, robots.txt.
- Fix blocks and request re-indexing.
rea-analysis-optimizationmorluto/reaOptimize REA providers and shared analysis workflows for agent context, latency, CPU, and memory while preserving correctness and Evidence. Use for analysis bottlenecks and cache or query design; CI/test-suite optimization belongs to rea-ci-tests-audit.Stars: 472kUpdated Claude CodeCodexVerifiedInstall
rea-ci-tests-auditmorluto/reaAudit REA CI and test suites for correctness, fixture drift, false passes, and measured performance; use for suite or workflow investigations and efficiency changes, rather than ordinary feature testing or a single test failure.Stars: 472kUpdated Claude CodeCodexVerifiedInstall
rea-tool-designmorluto/reaAudit, investigate, design, or change REA investigation tools, CLI/MCP contracts, provider capabilities, and Evidence semantics.Stars: 472kUpdated Claude CodeCodexVerifiedInstall
ponytail-auditDietrichGebert/ponytailQuality audit of the whole repo: bugs, security, real load, missing tests, speed, and what to delete. Most important first.Stars: 194kUpdated Claude CodeCodexVerifiedInstall