aiskillstore/marketplace / skills/aaron-he-zhu/technical-seo-checker
technical-seo-checker
VerifiedUse when the user asks to "check technical SEO"; audits crawlability, indexing, Core Web Vitals, robots.txt, sitemaps, canonicals, redirects, and migrations. Not for on-page tags or content — use on-page-seo-auditor. 技术SEO/网站速度
| File | Kind | Size |
|---|---|---|
| SKILL.md | skill md | 12.1 KB |
| references/bulk-audit-playbook.md | markdown | 4.2 KB |
| references/ecommerce-platform-patterns.md | markdown | 3.7 KB |
| references/http-status-codes.md | markdown | 2.5 KB |
| references/llm-crawler-handling.md | markdown | 4.5 KB |
| references/pre-migration-playbook.md | markdown | 6.0 KB |
| references/robots-txt-reference.md | markdown | 3.1 KB |
| references/technical-audit-example.md | markdown | 4.1 KB |
| references/technical-audit-templates.md | markdown | 6.2 KB |
| skill-report.json | config | 64.6 KB |
references/robots-txt-reference.md
View on GitHub# Robots.txt Reference Guide
Use robots.txt to control crawling, not indexing. To prevent indexing, use `noindex` meta or `X-Robots-Tag`.
## Directives
| Directive | Use | Notes |
|-----------|-----|-------|
| `User-agent: *` | All bots | Group consecutive user-agent lines before rules |
| `Disallow: /path/` | Block crawl path | `/admin/` is safer than `/admin` for directories |
| `Allow: /path/` | Override a broader block | Supported by Google, Bing, most major crawlers |
| `Sitemap: https://example.com/sitemap.xml` | Declare XML sitemap | Absolute URL; multiple lines allowed |
| `Crawl-delay: 10` | Slow some crawlers | Googlebot ignores it; use Search Console for Google |
## Common User Agents
`Googlebot`, `Bingbot`, `DuckDuckBot`, `OAI-SearchBot`, `GPTBot`, `ChatGPT-User`, `ClaudeBot`, `anthropic-ai`, `PerplexityBot`, `Perplexity-User`, `CCBot`, `Google-Extended`.
## AI Crawler Patterns
Block AI training and broad dataset crawlers while allowing search indexing and selected AI retrieval bots:
```txt
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: anthropic-ai
User-agent: CCBot
User-agent: Google-Extended
Disallow: /
User-agent: OAI-SearchBot
Disallow:
User-agent: ChatGPT-User
Disallow:
User-agent: PerplexityBot
User-agent: Perplexity-User
Disallow:
User-agent: Googlebot
Disallow:
User-agent: Bingbot
Disallow:
Sitemap: https://example.com/sitemap.xml
```
Allow only search engines: block `User-agent: *`, then explicitly allow Googlebot, Bingbot, DuckDuckBot, and required commercial crawlers.
## SEO-Critical Configs
| Scenario | Starter rules |
|----------|---------------|
| Parameter crawl waste | `Disallow: /*?` then `Allow: /?` only if homepage/search needs it |
| Ecommerce | Block account/cart/checkout/admin, parameter filters/sort/search; allow `/products/` |
| WordPress | Block `/wp-admin/`, allow `/wp-admin/admin-ajax.php`, allow uploads, block feeds/search/trackbacks when needed |
| Staging | `Disallow: /` plus noindex/auth; remove via Search Console if indexed |
## Mistakes and Fixes
| Mistake | Risk | Fix |
|---------|------|-----|
| Blocking CSS/JS | Google cannot render pages | Allow asset paths |
| Relative sitemap | May not parse | Use absolute sitemap URL |
| Spaces before colons | Invalid syntax | `User-agent: Googlebot` |
| Missing trailing slash | Over-blocks similar paths | Use `/admin/` for directory only |
| Using robots.txt for de-indexing | URL can still be indexed if linked | Use noindex/meta/header |
| Case mismatch | Paths are case-sensitive | Cover real URL variants |
## File Requirements
Returns 200, plain text UTF-8, located at `/robots.txt`, lowercase filename, under 500KB, tested in Search Console.
## Monitoring
Monthly: accessibility, blocked URLs, crawl stats. Quarterly: blocked paths, private sections, AI crawler changes. After migrations: retest URL structures and sitemap references.
## Emergency Fixes
If the site is accidentally blocked, change to `User-agent: *` plus empty `Disallow:`, include Sitemap, test in Search Console, and request recrawl. If CSS/JS is blocked, add asset `Allow` rules and re-render key URLs.
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| rea-analysis-optimizationmorluto/reaOptimize REA providers and shared analysis workflows for agent context, latency, CPU, and memory while preserving correctness and Evidence. Use for analysis bottlenecks and cache or query design; CI/test-suite optimization belongs to rea-ci-tests-audit. | Stars: 472k | Updated | Claude CodeCodex | Verified | Install |
| rea-ci-tests-auditmorluto/reaAudit REA CI and test suites for correctness, fixture drift, false passes, and measured performance; use for suite or workflow investigations and efficiency changes, rather than ordinary feature testing or a single test failure. | Stars: 472k | Updated | Claude CodeCodex | Verified | Install |
| rea-tool-designmorluto/reaAudit, investigate, design, or change REA investigation tools, CLI/MCP contracts, provider capabilities, and Evidence semantics. | Stars: 472k | Updated | Claude CodeCodex | Verified | Install |
| ponytail-auditDietrichGebert/ponytailQuality audit of the whole repo: bugs, security, real load, missing tests, speed, and what to delete. Most important first. | Stars: 194k | Updated | Claude CodeCodex | Verified | Install |