Basinfy/Basinfy-SaaS-Studio / .agents/skills/saas-seo-agentic
saas-seo-agentic
UnverifiedAudit & implement IsItAgentReady Level 0-5 standards (RFC 9727, llms.txt, MCP cards, JWKS).
| File | Kind | Size |
|---|---|---|
| SKILL.md | skill md | 6.0 KB |
| references/access-policy.md | markdown | 6.1 KB |
| references/agent-auth-and-identity.md | markdown | 3.6 KB |
| references/agent-friendly-pages.md | markdown | 5.1 KB |
| references/agentic-commerce-protocols.md | markdown | 2.2 KB |
| references/discovery-and-markdown.md | markdown | 6.7 KB |
| references/dns-aid-and-link-headers.md | markdown | 1.9 KB |
| references/isitagentready-levels.md | markdown | 4.2 KB |
| references/lighthouse-agentic-category.md | markdown | 6.3 KB |
| references/vendor-matrix.md | markdown | 5.0 KB |
| references/webmcp.md | markdown | 6.1 KB |
references/access-policy.md
View on GitHub# Agent access policy: robots.txt, Content-Signal, WAF, Web Bot Auth
Facts checked 2026-09-23 against vendor documentation unless marked
secondary. robots.txt is a **preference signal, not access control**: several
user-triggered agents do not treat it as binding, and TollBit's State of the
Bots reports (secondary) found a meaningful share of AI fetchers reaching
disallowed URLs. Protect private paths with authentication.
## User-agent tokens by purpose
Check the bot that governs the claim you are making. Crawler roles and the
citability rules live in `seo-geo`; this table adds what matters for agents.
| Token | Vendor | Purpose | robots.txt (vendor docs) | Verify by |
|---|---|---|---|---|
| GPTBot | OpenAI | model training | honoured | openai.com/gptbot.json |
| OAI-SearchBot | OpenAI | ChatGPT search index | honoured | openai.com/searchbot.json |
| ChatGPT-User | OpenAI | user-initiated fetches and actions | "may not apply" | openai.com/chatgpt-user.json; Web Bot Auth |
| OAI-AdsBot | OpenAI | ad landing-page checks | see vendor docs | openai.com/adsbot.json |
| ClaudeBot | Anthropic | model training | honoured (incl. Crawl-delay) | claude.com/crawling/bots.json |
| Claude-SearchBot | Anthropic | Claude search-result citability | honoured | claude.com/crawling/bots.json |
| Claude-User | Anthropic | user-initiated fetches | honoured | claude.com/crawling/bots.json |
| PerplexityBot | Perplexity | search index | honoured | perplexity.com/perplexitybot.json |
| Perplexity-User | Perplexity | user-initiated fetches | generally ignored | perplexity.com/perplexity-user.json |
| Google-Agent | Google | user-triggered agent fetches | generally ignored | developers.google.com/static/crawling/ipranges/user-triggered-agents.json |
| Google-Extended | Google | Gemini training/grounding (and Search gen-AI model training) control token, no crawler; never Search inclusion | honoured | n/a |
Anthropic's help center previously said IP ranges were not published; as of
2026-08 they are at `claude.com/crawling/bots.json`. Do not IP-block
Anthropic's crawlers: a blocked crawler cannot read robots.txt at all.
## RFC 9309 group selection (the most common mistake)
A crawler obeys **only** the most specific group that names its token. If
any group names it, the `*` group is ignored entirely for that crawler. So a
line placed in `User-agent: *` does not reach `GPTBot` when a
`User-agent: GPTBot` group exists.
This matters for `Content-Signal`. Cloudflare's managed robots.txt puts it in
the `*` group, and no normative text says it escapes group selection. When a
site has named groups, repeat the line inside each one.
`agentic_fix.py robots` does exactly that, and never touches Allow/Disallow.
A 5xx on robots.txt tells compliant crawlers to treat the site as fully
disallowed. A 4xx means "no restrictions".
## Content-Signal
Syntax: `Content-Signal: search=yes, ai-input=yes, ai-train=no`.
| Key | Meaning |
|---|---|
| `search` | Building a search index and showing links/snippets (excludes AI summaries) |
| `ai-input` | Using content as input to an AI answer (RAG, grounding) |
| `ai-train` | Training or fine-tuning models |
| `use` | Experimental Cloudflare field (`immediate`, `reference`, `full`); Cloudflare's managed default includes `use=reference` (docs give no date) |
Status: Cloudflare's CC0 Content Signals Policy (launched 2025-09-24). The
IETF individual draft `draft-romm-aipref-contentsignals` expired 2026-04-04;
the IETF AIPREF working group vocabulary (`draft-ietf-aipref-vocab`) has not
reached consensus. We found no Google statement about Content-Signal; Google's
robots.txt spec lists only the fields it supports, and this is not one. Present
it as a stated preference with no confirmed effect.
## Illustrative robots.txt (welcomes agents and search, not training)
```
User-agent: *
Content-Signal: search=yes, ai-input=yes, ai-train=no
Allow: /
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
User-agent: Perplexity-User
Content-Signal: search=yes, ai-input=yes, ai-train=no
Allow: /
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
Content-Signal: search=yes, ai-input=yes, ai-train=no
Disallow: /
Sitemap: https://example.com/sitemap.xml
```
Blocking a training crawler is a business decision. Never recommend it by
default, and never present it as affecting Google Search. Google-Agent needs no
line because it generally ignores robots.txt; put `/checkout/` and
`/account/` behind authentication.
## WAF and bot management
- Allowlist **verified** bots and signed agents, not user-agent strings.
Cloudflare, AWS WAF, Akamai, HUMAN and Vercel verify Web Bot Auth or
published IP ranges; Perplexity documents WAF recipes.
- Keep CAPTCHAs and challenge pages off content and informational pages.
- A WAF that challenges an unverified request carrying `GPTBot` is behaving
correctly. `agentic_check.py --ua-matrix` shows how unverified traffic is
treated; confirm real-agent behaviour in WAF logs.
## Web Bot Auth
How agents prove identity: HTTP Message Signatures (RFC 9421), Ed25519 keys,
a key directory at `/.well-known/http-message-signatures-directory`
(`application/http-message-signatures-directory+json`), and `tag="web-bot-auth"`.
- Current draft: `draft-ietf-webbotauth-httpsig-protocol-00` (2026-09-01),
replacing the earlier Meunier individual drafts.
- `Signature-Agent` is now a **dictionary keyed by signature label**, e.g.
`Signature-Agent: sig1="https://signer.example"`. The bare-string form is
outdated; some verifiers may still expect it, so test before relying on it.
- Signers reported: OpenAI (agent and cloud-browser traffic), Google
(experimental, `https://agent.bot.goog`), Amazon Bedrock AgentCore, Shopify.
Anthropic and Perplexity signing is claimed only by secondary sources.
Site owners do not publish a key directory unless they operate an agent. They
verify signatures at the edge.
## Reporting rules
- Report training, search and user-fetch access on separate lines.
- Label Content-Signal and Web Bot Auth as draft or policy, with the check
date.
- Never promise ranking, citation or traffic effects from any robots change.
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| rea-analysis-optimizationmorluto/reaOptimize REA providers and shared analysis workflows for agent context, latency, CPU, and memory while preserving correctness and Evidence. Use for analysis bottlenecks and cache or query design; CI/test-suite optimization belongs to rea-ci-tests-audit. | Stars: 472k | Updated | Claude CodeCodex | Verified | Install |
| rea-ci-tests-auditmorluto/reaAudit REA CI and test suites for correctness, fixture drift, false passes, and measured performance; use for suite or workflow investigations and efficiency changes, rather than ordinary feature testing or a single test failure. | Stars: 472k | Updated | Claude CodeCodex | Verified | Install |
| rea-tool-designmorluto/reaAudit, investigate, design, or change REA investigation tools, CLI/MCP contracts, provider capabilities, and Evidence semantics. | Stars: 472k | Updated | Claude CodeCodex | Verified | Install |
| ponytail-auditDietrichGebert/ponytailQuality audit of the whole repo: bugs, security, real load, missing tests, speed, and what to delete. Most important first. | Stars: 194k | Updated | Claude CodeCodex | Verified | Install |