ericrisco/rsc-harness / skills/seo-geo
seo-geo
VerifiedUse when one existing page needs to rank in Google AND get cited by AI answer engines — auditing a URL for on-page SEO, structured-data JSON-LD, GEO citation levers, Core Web Vitals, crawlability, and AI-bot access in robots.txt. NOT writing the article body (that is `article-writing`), NOT keyword research or content strategy (that is `content-engine`).
| File | Kind | Size |
|---|---|---|
| SKILL.md | skill md | 11.1 KB |
| evals/README.md | markdown | 1.0 KB |
| evals/cases.yaml | config | 3.7 KB |
| references/ai-crawler-control.md | markdown | 3.6 KB |
| references/schema-recipes.md | markdown | 4.3 KB |
| scripts/verify.shexecutable | script | 9.1 KB |
references/ai-crawler-control.md
View on GitHub# AI-crawler control (robots.txt, 2026)
AI access to your site is governed by **robots.txt + real on-page authority** — not by
`llms.txt` (Google ignores it; AI bots skip it). The lever that matters is *which
user-agent you allow*, because vendors split their crawlers **by purpose**, and blocking
the wrong one silently removes you from an engine's citations.
## The purpose split
| User-agent | Vendor | Purpose | Block effect |
|---|---|---|---|
| `GPTBot` | OpenAI | Training corpus | Out of training; **still citable** |
| `OAI-SearchBot` | OpenAI | ChatGPT Search index | **Out of ChatGPT Search citations** |
| `ChatGPT-User` | OpenAI | Live user-triggered fetch | No live fetch on user action |
| `ClaudeBot` | Anthropic | Training corpus | Out of training; still citable |
| `Claude-SearchBot` | Anthropic | Search index | Out of Claude search citations |
| `Claude-User` | Anthropic | Live user-triggered fetch | No live fetch |
| `PerplexityBot` | Perplexity | Search index | **Out of Perplexity citations** |
| `Google-Extended` | Google | Gemini/Vertex training opt-out | Out of Google AI training |
| `Googlebot` | Google | Search crawl + **AI Overviews** | Out of Search entirely (see caveat) |
| `Applebot-Extended` | Apple | Apple AI training opt-out | Out of Apple AI training |
| `CCBot` | Common Crawl | Open dataset (feeds many models) | Out of Common Crawl |
| `meta-externalagent` | Meta | Meta AI crawl | Out of Meta AI |
## Caveats that bite
- **Google AI Overviews ride ordinary Googlebot.** `Google-Extended` opts you out of
Gemini *training* only. There is **no way to opt out of AI Overviews without blocking
Googlebot — i.e. leaving Google Search.** Do not promise a client "AI Overviews opt-out."
- **Search bot ≠ training bot.** "We blocked GPTBot" does **not** remove you from ChatGPT
Search citations — that is `OAI-SearchBot`. People routinely block the wrong one.
- **Perplexity circumvention.** Cloudflare reported (Aug 2025) that Perplexity fetched
pages via undeclared agents after being disallowed. robots.txt is an honor-system
directive; enforce hard blocks with a WAF/firewall, not just robots.txt, if you truly
must keep a bot out.
## Template — "cite me, but don't train on me"
Keep search/citation bots in; keep training bots out.
```text
# robots.txt — citation-eligible, training opted out
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: CCBot
Disallow: /
User-agent: Google-Extended
Disallow: /
# Search/citation bots: ALLOWED (no Disallow = allowed)
User-agent: OAI-SearchBot
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: Googlebot
Allow: /
Sitemap: https://example.com/sitemap.xml
```
## Template — "block all AI"
Accepts loss of AI citations. Note this does NOT block Google AI Overviews (that needs
blocking Googlebot, which leaves Search).
```text
# robots.txt — block AI training AND AI search
User-agent: GPTBot
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: ClaudeBot
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
User-agent: CCBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: meta-externalagent
Disallow: /
# Googlebot left ALLOWED — blocking it would remove you from Search.
User-agent: Googlebot
Allow: /
Sitemap: https://example.com/sitemap.xml
```
## Audit check
If the brief says "we want AI citations," confirm robots.txt does **not** `Disallow: /`
under `OAI-SearchBot`, `PerplexityBot`, `Claude-SearchBot`, or `Googlebot`. A self-blocking
robots.txt is the most common reason a "GEO-optimized" page earns zero citations.
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| rea-analysis-optimizationmorluto/reaOptimize REA providers and shared analysis workflows for agent context, latency, CPU, and memory while preserving correctness and Evidence. Use for analysis bottlenecks and cache or query design; CI/test-suite optimization belongs to rea-ci-tests-audit. | Stars: 472k | Updated | Claude CodeCodex | Verified | Install |
| rea-ci-tests-auditmorluto/reaAudit REA CI and test suites for correctness, fixture drift, false passes, and measured performance; use for suite or workflow investigations and efficiency changes, rather than ordinary feature testing or a single test failure. | Stars: 472k | Updated | Claude CodeCodex | Verified | Install |
| rea-tool-designmorluto/reaAudit, investigate, design, or change REA investigation tools, CLI/MCP contracts, provider capabilities, and Evidence semantics. | Stars: 472k | Updated | Claude CodeCodex | Verified | Install |
| ponytail-auditDietrichGebert/ponytailQuality audit of the whole repo: bugs, security, real load, missing tests, speed, and what to delete. Most important first. | Stars: 194k | Updated | Claude CodeCodex | Verified | Install |