Skip to content

ericrisco/rsc-harness / skills/seo-geo

seo-geo

Verified

Use when one existing page needs to rank in Google AND get cited by AI answer engines — auditing a URL for on-page SEO, structured-data JSON-LD, GEO citation levers, Core Web Vitals, crawlability, and AI-bot access in robots.txt. NOT writing the article body (that is `article-writing`), NOT keyword research or content strategy (that is `content-engine`).

FileKindSize
SKILL.mdskill md11.1 KB
evals/README.mdmarkdown1.0 KB
evals/cases.yamlconfig3.7 KB
references/ai-crawler-control.mdmarkdown3.6 KB
references/schema-recipes.mdmarkdown4.3 KB
scripts/verify.shexecutablescript9.1 KB

references/ai-crawler-control.md

View on GitHub
# AI-crawler control (robots.txt, 2026)

AI access to your site is governed by **robots.txt + real on-page authority** — not by
`llms.txt` (Google ignores it; AI bots skip it). The lever that matters is *which
user-agent you allow*, because vendors split their crawlers **by purpose**, and blocking
the wrong one silently removes you from an engine's citations.

## The purpose split

| User-agent | Vendor | Purpose | Block effect |
|---|---|---|---|
| `GPTBot` | OpenAI | Training corpus | Out of training; **still citable** |
| `OAI-SearchBot` | OpenAI | ChatGPT Search index | **Out of ChatGPT Search citations** |
| `ChatGPT-User` | OpenAI | Live user-triggered fetch | No live fetch on user action |
| `ClaudeBot` | Anthropic | Training corpus | Out of training; still citable |
| `Claude-SearchBot` | Anthropic | Search index | Out of Claude search citations |
| `Claude-User` | Anthropic | Live user-triggered fetch | No live fetch |
| `PerplexityBot` | Perplexity | Search index | **Out of Perplexity citations** |
| `Google-Extended` | Google | Gemini/Vertex training opt-out | Out of Google AI training |
| `Googlebot` | Google | Search crawl + **AI Overviews** | Out of Search entirely (see caveat) |
| `Applebot-Extended` | Apple | Apple AI training opt-out | Out of Apple AI training |
| `CCBot` | Common Crawl | Open dataset (feeds many models) | Out of Common Crawl |
| `meta-externalagent` | Meta | Meta AI crawl | Out of Meta AI |

## Caveats that bite

- **Google AI Overviews ride ordinary Googlebot.** `Google-Extended` opts you out of
  Gemini *training* only. There is **no way to opt out of AI Overviews without blocking
  Googlebot — i.e. leaving Google Search.** Do not promise a client "AI Overviews opt-out."
- **Search bot ≠ training bot.** "We blocked GPTBot" does **not** remove you from ChatGPT
  Search citations — that is `OAI-SearchBot`. People routinely block the wrong one.
- **Perplexity circumvention.** Cloudflare reported (Aug 2025) that Perplexity fetched
  pages via undeclared agents after being disallowed. robots.txt is an honor-system
  directive; enforce hard blocks with a WAF/firewall, not just robots.txt, if you truly
  must keep a bot out.

## Template — "cite me, but don't train on me"

Keep search/citation bots in; keep training bots out.

```text
# robots.txt — citation-eligible, training opted out

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: Google-Extended
Disallow: /

# Search/citation bots: ALLOWED (no Disallow = allowed)
User-agent: OAI-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: Googlebot
Allow: /

Sitemap: https://example.com/sitemap.xml
```

## Template — "block all AI"

Accepts loss of AI citations. Note this does NOT block Google AI Overviews (that needs
blocking Googlebot, which leaves Search).

```text
# robots.txt — block AI training AND AI search

User-agent: GPTBot
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: ClaudeBot
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
User-agent: CCBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: meta-externalagent
Disallow: /

# Googlebot left ALLOWED — blocking it would remove you from Search.
User-agent: Googlebot
Allow: /

Sitemap: https://example.com/sitemap.xml
```

## Audit check

If the brief says "we want AI citations," confirm robots.txt does **not** `Disallow: /`
under `OAI-SearchBot`, `PerplexityBot`, `Claude-SearchBot`, or `Googlebot`. A self-blocking
robots.txt is the most common reason a "GEO-optimized" page earns zero citations.
rea-analysis-optimizationmorluto/reaOptimize REA providers and shared analysis workflows for agent context, latency, CPU, and memory while preserving correctness and Evidence. Use for analysis bottlenecks and cache or query design; CI/test-suite optimization belongs to rea-ci-tests-audit.Stars: 472kUpdated Claude CodeCodexVerifiedInstall
rea-ci-tests-auditmorluto/reaAudit REA CI and test suites for correctness, fixture drift, false passes, and measured performance; use for suite or workflow investigations and efficiency changes, rather than ordinary feature testing or a single test failure.Stars: 472kUpdated Claude CodeCodexVerifiedInstall
rea-tool-designmorluto/reaAudit, investigate, design, or change REA investigation tools, CLI/MCP contracts, provider capabilities, and Evidence semantics.Stars: 472kUpdated Claude CodeCodexVerifiedInstall
ponytail-auditDietrichGebert/ponytailQuality audit of the whole repo: bugs, security, real load, missing tests, speed, and what to delete. Most important first.Stars: 194kUpdated Claude CodeCodexVerifiedInstall