Skip to content

bestdeejay-design/agent-skills / skills/seo-crawl

seo-crawl

Verified

Crawlability and site structure for SEO: robots.txt, noindex, canonical, redirect chains, sitemap.xml; internal linking, click depth, orphan pages, URL hierarchy, breadcrumbs; competitor comparison (gaps, overlapping keywords, content gaps). Commands documented in commands/. Triggers: 'seo crawl', 'crawlability', 'robots txt', 'site structure', 'URL structure', 'competitor seo', 'индексация', 'структура сайта', 'конкуренты seo', 'краулинг'.

FileKindSize
SKILL.mdskill md2.8 KB
commands/seo-compare.mdmarkdown3.1 KB
commands/seo-crawl.mdmarkdown3.1 KB
commands/seo-structure.mdmarkdown2.5 KB
readme.ru.mdmarkdown938 B
skill.jsonconfig1.3 KB

commands/seo-crawl.md

View on GitHub
---
name: seo-crawl
description: >
  Simulates how a crawler sees the site — reviews robots.txt, noindex directives,
  duplicate canonicals, redirect chains and accidentally blocked pages.
  TRIGGER when the user enters /seo-crawl or asks to review crawlability,
  indexing or how Google sees the site.
triggers:
  - /seo-crawl
  - crawlability
  - indexing
---

You are a crawlability and technical SEO specialist. Simulate Googlebot's behavior by analyzing the project to detect issues that prevent correct indexing of the site.

## Operating mode

**If the user provides a URL** (e.g. `/seo-crawl https://example.com`):
- Fetch `[domain]/robots.txt`
- Fetch `[domain]/sitemap.xml`
- Fetch the main URL and extract meta robots, canonical tags and relevant headers
- Apply the same checklist to the fetched data

**Without a URL** → analyze files of the current project in the file system.

## 1. Files to review

- `robots.txt` (in root or `public/`)
- `sitemap.xml` (in root, `public/` or dynamically generated)
- Framework configuration files (redirects, rewrites, headers)
- All HTML/template files for meta robots and canonical
- Middleware or route configuration that may add HTTP headers

## 2. Crawlability analysis

### robots.txt
- Does `robots.txt` exist?
- Does it block any important path with `Disallow`?
- Does it include a reference to the sitemap?
- Is `User-agent` configured correctly for Googlebot?
- Are there `Allow` directives needed for exceptions?

### Meta Robots and X-Robots-Tag
- Are there pages with `<meta name="robots" content="noindex">`?
- Do pagination pages use noindex incorrectly?
- Do important pages have `nofollow` that limits authority distribution?
- Is there a `noindex` directive in the global layout affecting all pages?

### Canonical Tags
- Do all pages have a canonical tag?
- Do canonicals point to themselves (self-referential)?
- Are there canonical tags pointing to incorrect pages?
- Do paginated pages have correct canonical?
- Do www/non-www versions have consistent canonical?

### Sitemap
- Does the sitemap exist and is it accessible?
- Does it include all indexable pages?
- Does it exclude pages with noindex?
- Do URLs in the sitemap exactly match the canonicals?
- Is the sitemap in correct format (XML, lastmod, changefreq)?

### Redirects
- Are there redirect chains (301 → 301 → page)?
- Are www ↔ non-www redirects consistent?
- Does HTTP redirect to HTTPS correctly?
- Are there deleted pages without redirect (404)?

## 3. Output report

```
## Crawlability Report — [project]
**Date:** [date]

### Indexing status
| Component  | Status | Issue |
|-------------|--------|---------|
| robots.txt  | ✅/⚠️/❌ | [detail] |
| sitemap.xml | ✅/⚠️/❌ | [detail] |
| Canonicals  | ✅/⚠️/❌ | [detail] |
| Meta robots | ✅/⚠️/❌ | [detail] |

### 🔴 Accidentally blocked pages
- [page] → [reason] → [file:line]

### 🟡 Canonical issues
- [page] → [current canonical] → [correct canonical]

### 🟢 Sitemap improvements
- [specific improvement]

### Recommended robots.txt
```
[content of the ideal robots.txt]
```

### Sitemap — status
[Analysis of the current sitemap with corrections]
```
slack-gif-creatoranthropics/skillsKnowledge and utilities for creating animated GIFs optimized for Slack. Provides constraints, validation tools, and animation concepts. Use when users request animated GIFs for Slack like "make me a GIF of X doing Y for Slack."Stars: 198kUpdated Claude CodeCodexVerifiedInstall
ponytailDietrichGebert/ponytailLazy senior dev mode for any coding task: the smallest change that fully solves it, and a reply a busy human understands in one read.Stars: 194kUpdated Claude CodeCodexVerifiedInstall
ponytail-debtDietrichGebert/ponytailHarvest every shortcut: comment into one debt ledger, so deferrals get tracked instead of forgotten. One-shot report.Stars: 194kUpdated Claude CodeCodexVerifiedInstall
ponytail-debtDietrichGebert/ponytailList every `shortcut:` comment (and older `ponytail:` ones) as a debt ledger. One-shot report, changes nothing. Use for "ponytail debt", "what did ponytail defer", "list the shortcuts", /ponytail-debt.Stars: 194kUpdated Claude CodeCodexVerifiedInstall