laguagu/claude-code-nextjs-skills / skills/nextjs-seo
nextjs-seo
VerifiedNext.js App Router SEO implementation and audits. Use for metadata and social images, sitemap/robots, canonicals and localization, structured data, Core Web Vitals evidence, AI crawler policy or search-indexing diagnosis. Not for general Next.js feature work unrelated to SEO.
| File | Kind | Size |
|---|---|---|
| SKILL.md | skill md | 4.0 KB |
| references/ai-search.md | markdown | 7.1 KB |
| references/checklist.md | markdown | 2.6 KB |
| references/json-ld.md | markdown | 2.2 KB |
| references/metadata-api.md | markdown | 3.7 KB |
| references/sitemap-robots.md | markdown | 2.9 KB |
| references/troubleshooting.md | markdown | 4.3 KB |
references/ai-search.md
View on GitHub# AI search and crawler controls
## Contents
- Eligibility and evidence
- Useful content without filler
- Crawlers and policy
- robots.ts example
- llms.txt
- Measurement and sources
## Eligibility and evidence
Google says AI Overviews and AI Mode need no special schema or AI text file.
The page must be indexed and eligible for a Search snippet. This is eligibility,
not a guarantee of selection or citation. Apply ordinary technical SEO first.
Check the effective Search generative AI inclusion setting (see below) when
diagnosing absence from Google's AI features.
Do not describe FAQPage, a TL;DR, llms.txt, an author bio or a particular word
count as a demonstrated AI ranking factor. Separate vendor-documented behavior
from hypotheses.
## Useful content without filler
Answer the actual question directly. Use descriptive headings and original,
verifiable information. Add a Q&A section only for useful questions that the
page has not already answered. Do not repeat the hero in a summary, FAQ and
keyword paragraph. There is no required opening word count.
A visible update date helps when freshness matters, but must track meaningful
changes. Keep the primary answer available in server-rendered HTML where
practical; crawler JavaScript support varies. Clear HTML is more portable than
relying on client-only fetching.
## Crawlers and policy
Check current vendor docs before changing robots rules. Training, search and
user-initiated fetching are different purposes; a vendor may use separate bots
or a product-control token. Preserve the owner's policy instead of copying a
blanket allow/block list.
A `User-Agent: __usage_ai-training` group (individual IETF draft
`draft-kelley-aipref-training-preference-signal`, based on aipref working-group
work) states a per-path preference about AI-training use. It does not restrict
crawling, and a `User-Agent: *` group does not count as such a preference.
Anthropic honors it, and a group naming ClaudeBot takes priority over it. Check
whether other vendors honor it before relying on it; their documented training
tokens (GPTBot, Google-Extended) still apply.
| Vendor | Training | Search | User-initiated retrieval |
|--------|----------|--------|--------------------------|
| OpenAI | GPTBot | OAI-SearchBot | ChatGPT-User |
| Anthropic | ClaudeBot | Claude-SearchBot | Claude-User |
| Perplexity | None (PerplexityBot is not used for training) | PerplexityBot | Perplexity-User |
Google Search and its AI features use Googlebot controls; `noindex`, `nosnippet`,
`data-nosnippet` and `max-snippet` limit what they can show. `Google-Extended`
has no separate HTTP user agent: it is a robots product token controlling both
Gemini training and certain Gemini/Vertex AI grounding uses. It does not control
Google Search inclusion or ranking. Do not call it a training-only switch or
promise that blocking it leaves Gemini grounding and its referrals unaffected.
Search Console's [Search generative AI control](https://support.google.com/webmasters/answer/16908024) (Settings > Search generative AI)
offers include/exclude/inherit for AI Overviews, AI Mode and Discover AI features; top-level properties default to include, child properties inherit.
It controls applicable AI-feature inclusion, not AI training or inclusion/ranking elsewhere in Search.
Blocking a search crawler can prevent direct content retrieval; it does not
prove that a URL or brand can never be mentioned through other sources.
User-initiated fetchers differ: OpenAI says robots.txt "may not apply" to
ChatGPT-User, Perplexity-User generally ignores it, and Anthropic says all three
of its bots honor it. Anthropic's hosted agent browsers are separate from those
bots: they send a normal browser User-Agent with a `Claude-Agent` token,
Anthropic manages them apart from its robots and does not say robots.txt
applies, and their traffic is verified against the IPs listed under
`Claude-Agent` at `https://claude.com/.well-known/sail`. Each vendor's settings
are independent, so allowing OAI-SearchBot while disallowing GPTBot is a valid
policy.
Inspect hosting firewall rules separately when diagnosing denied public crawls. A
User-Agent string can be spoofed; use vendor-documented verification methods
for bot identity, and do not weaken preview protection to make an audit pass.
## robots.ts example
Start with the existing wildcard policy. Add named groups only when a deliberate
exception is needed. Specific groups do not inherit wildcard restrictions.
```typescript
import type { MetadataRoute } from 'next';
export default function robots(): MetadataRoute.Robots {
const disallow = ['/api/', '/admin/'];
return {
rules: [
{ userAgent: '*', allow: '/', disallow },
// Only if the owner chooses to opt out of OpenAI training:
// { userAgent: 'GPTBot', disallow: '/' },
// If a named search group is needed, repeat the restrictions:
// { userAgent: 'OAI-SearchBot', allow: '/', disallow },
],
sitemap: 'https://your-site.com/sitemap.xml',
};
}
```
## llms.txt
Treat llms.txt as an optional documentation-discovery convention, not a proven
citation boost. Google Search does not use it, so it neither helps nor hurts
there. It can be useful when developers or
agents explicitly consume documentation through it. Verify the intended consumer
before adding it. Prefer a static `public/llms.txt` for a fixed document; use a
Route Handler only when generation is needed, with caching supported by the
installed Next.js configuration. Do not copy `dynamic = 'force-static'` into a
Cache Components app.
## Measurement and sources
Use referral analytics and concrete citation examples to observe changes.
Missing referrals do not prove missing citations; crawler hits do not prove
citations. Google's Search Console includes AI-feature traffic in overall Web
search reporting, so do not label the whole export as AI traffic.
Search Console's Generative AI performance report shows
AI Overviews and AI Mode link impressions by page, country, date and device.
If it is unavailable, check impression volume and the effective inclusion
setting; insufficient impressions or exclusion are possible causes.
Sources to re-check for changing policies:
- [Google AI features](https://developers.google.com/search/docs/appearance/ai-features)
- [Google AI optimization guide](https://developers.google.com/search/docs/fundamentals/ai-optimization-guide)
- [Generative AI performance report (Search)](https://support.google.com/webmasters/answer/16984139)
- [Google-Extended](https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers#google-extended)
- [OpenAI crawlers](https://developers.openai.com/api/docs/bots)
- [Anthropic crawlers](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler)
- [Anthropic agent browsers](https://support.claude.com/en/articles/18234133-how-do-anthropic-hosted-browsers-identify-themselves-when-they-browse-the-web-on-a-user-s-behalf)
- [Training preference signal draft](https://datatracker.ietf.org/doc/draft-kelley-aipref-training-preference-signal/)
- [Perplexity crawlers](https://docs.perplexity.ai/guides/bots)
- [llms.txt proposal](https://llmstxt.org/)
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| rea-analysis-optimizationmorluto/reaOptimize REA providers and shared analysis workflows for agent context, latency, CPU, and memory while preserving correctness and Evidence. Use for analysis bottlenecks and cache or query design; CI/test-suite optimization belongs to rea-ci-tests-audit. | Stars: 472k | Updated | Claude CodeCodex | Verified | Install |
| rea-ci-tests-auditmorluto/reaAudit REA CI and test suites for correctness, fixture drift, false passes, and measured performance; use for suite or workflow investigations and efficiency changes, rather than ordinary feature testing or a single test failure. | Stars: 472k | Updated | Claude CodeCodex | Verified | Install |
| rea-tool-designmorluto/reaAudit, investigate, design, or change REA investigation tools, CLI/MCP contracts, provider capabilities, and Evidence semantics. | Stars: 472k | Updated | Claude CodeCodex | Verified | Install |
| ponytail-auditDietrichGebert/ponytailQuality audit of the whole repo: bugs, security, real load, missing tests, speed, and what to delete. Most important first. | Stars: 194k | Updated | Claude CodeCodex | Verified | Install |