Skip to content

aiskillstore/marketplace / skills/aaron-he-zhu/technical-seo-checker

technical-seo-checker

Verified

Use when the user asks to "check technical SEO"; audits crawlability, indexing, Core Web Vitals, robots.txt, sitemaps, canonicals, redirects, and migrations. Not for on-page tags or content — use on-page-seo-auditor. 技术SEO/网站速度

references/llm-crawler-handling.md

View on GitHub
# LLM Crawler Handling

Use robots.txt and server policy to decide which AI crawlers can access content. This is a policy and technical SEO decision, not only a crawler-control task.

## Crawler Matrix

| Bot | Operator / Use | Typical Rule |
|-----|----------------|--------------|
| OAI-SearchBot | OpenAI search/citation discovery | Allow when ChatGPT search visibility is desired |
| GPTBot | OpenAI training/crawling | Allow if AI visibility/data use is acceptable; block for TDM restriction |
| ChatGPT-User | OpenAI user-requested browsing/actions | Usually allow for user-triggered access; robots.txt may not apply |
| ClaudeBot / anthropic-ai | Anthropic crawling/training | Same policy decision as GPTBot |
| Claude-SearchBot | Anthropic search/citation discovery for Claude | Allow when Claude search-citation visibility is desired (the retrieval analog of OAI-SearchBot) |
| Claude-User | Anthropic user-triggered browsing/actions | Usually allow for user-triggered access; robots.txt may not apply |
| Google-Extended | Google AI training opt-out | Block to restrict training while Googlebot can still index. Does not affect AI Overviews / AI Mode, which use the Google index — opting out of those is not available via robots.txt |
| Googlebot | Search indexing | Usually allow |
| Bingbot | Search indexing / Copilot ecosystem | Usually allow |
| PerplexityBot | AI answer retrieval | Allow when citation visibility is desired |
| Perplexity-User | Perplexity user-triggered fetcher | Allow only if logs/IP docs confirm this access path is wanted |
| CCBot | Common Crawl | Block when broad dataset reuse is not desired |

## Policy Modes

| Mode | Use When | Robots Pattern |
|------|----------|----------------|
| default-open | AI visibility and citation discovery are goals | Allow search, retrieval, and selected AI bots; block only sensitive paths |
| default-closed | Licensed, paid, private, or TDM-reserved content | Block broad AI crawlers by default; allow only approved search/retrieval bots |
| split | Search indexing yes, AI training no | Allow Googlebot/Bingbot/OAI-SearchBot/Claude-SearchBot/selected retrieval bots; block GPTBot, ClaudeBot, CCBot, Google-Extended |

## Search-Only Starter

```txt
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: anthropic-ai
User-agent: CCBot
User-agent: Google-Extended
Disallow: /

User-agent: OAI-SearchBot
Disallow:

User-agent: ChatGPT-User
Disallow:

User-agent: PerplexityBot
User-agent: Perplexity-User
Disallow:

User-agent: Claude-SearchBot
User-agent: Claude-User
Disallow:

User-agent: Googlebot
User-agent: Bingbot
Disallow:

Sitemap: https://example.com/sitemap.xml
```

> **Note**: this "search-only" pattern blocks `ClaudeBot` (training) but explicitly allows
> `Claude-SearchBot`/`Claude-User` — otherwise you silently lose Claude's search citations while
> only meaning to opt out of training. Mirror the same allow/block split for every assistant.

## Technical Checks

| Check | Why |
|-------|-----|
| robots.txt returns 200 and parses | Crawler policy must be readable |
| Search bots still allowed | Avoid accidental SEO loss |
| Sitemap references current canonical URLs | Supports discovery |
| Retrieval bots include OAI-SearchBot, ChatGPT-User, PerplexityBot, Perplexity-User, Claude-SearchBot, and Claude-User where desired | Prevents accidental citation loss |
| Published IP ranges match provider JSON when edge rules are used | Avoids spoofing and stale allowlists |
| Private/gated paths use auth or noindex, not only robots.txt | robots.txt is not access control |
| Logs confirm bot behavior | Validate crawl policy after launch |

## Cloudflare Edge-Override Gotcha

Cloudflare and other edge tools can override origin robots.txt, block user agents before they reach the file, or serve different rules by host/path. Check WAF/bot rules, Workers, Transform Rules, cache variants, and raw origin response before concluding the published robots.txt is the effective crawler policy.

## Legal/Compliance Notes

EU AI Act Art 53 and EU DSM TDM reservations may matter for rights-reserved content. Robots.txt can signal intent, but it is not a complete training opt-out or licensing mechanism. For regulated/licensed content, pair crawler policy with contracts, access controls, rights-reservation notices (for example TDM reservation where adopted), and any `X-Robots-Tag` directives your target crawlers actually honor.

## Reporting Fields

Policy mode, allowed bots, blocked bots, affected paths, business rationale, source/date, expected SEO/GEO effect, and retest date.
rea-analysis-optimizationmorluto/reaOptimize REA providers and shared analysis workflows for agent context, latency, CPU, and memory while preserving correctness and Evidence. Use for analysis bottlenecks and cache or query design; CI/test-suite optimization belongs to rea-ci-tests-audit.Stars: 472kUpdated Claude CodeCodexVerifiedInstall
rea-ci-tests-auditmorluto/reaAudit REA CI and test suites for correctness, fixture drift, false passes, and measured performance; use for suite or workflow investigations and efficiency changes, rather than ordinary feature testing or a single test failure.Stars: 472kUpdated Claude CodeCodexVerifiedInstall
rea-tool-designmorluto/reaAudit, investigate, design, or change REA investigation tools, CLI/MCP contracts, provider capabilities, and Evidence semantics.Stars: 472kUpdated Claude CodeCodexVerifiedInstall
ponytail-auditDietrichGebert/ponytailQuality audit of the whole repo: bugs, security, real load, missing tests, speed, and what to delete. Most important first.Stars: 194kUpdated Claude CodeCodexVerifiedInstall