iamwaqargulzar/Marketing-Agent-OS / skills/brand-language-codifier
brand-language-codifier
VerifiedUse when the user asks to "codify our brand voice", "define naming rules for our products and tiers", or "write the tone-of-voice guide with banned phrases"; produces the brand-level voice canon (register, tone spectrum, banned-phrase list, few-shot examples drawn only from the brand's own published material) and the naming tax (product / feature / tier naming rules plus approved and banned terms) that seeds the narrative-registry canon and that every channel's voice adaptation points up to. Not for per-platform voice adaptation — use channel-registry's voice-dossier; not for finished copy or blog posts — use content-writer; not for the message hierarchy itself — use message-system-architect; not for claim adjudication — use offer-claims-registry. 品牌语气/词汇表/命名税/禁用词/品牌语言规范
| File | Kind | Size |
|---|---|---|
| LICENSE.txt | markdown | 6.7 KB |
| NOTICE.md | markdown | 325 B |
| SKILL.md | skill md | 12.6 KB |
| references/shared/skill-contract.md | markdown | 15.4 KB |
| references/shared/tale-benchmark.md | markdown | 8.6 KB |
| references/skill-contract.md | markdown | 2.4 KB |
| scripts/connectors/_http.py | script | 28.2 KB |
| scripts/connectors/_loader.py | script | 1.2 KB |
| scripts/connectors/firecrawl.py | script | 13.1 KB |
| scripts/connectors/robots.py | script | 12.8 KB |
scripts/connectors/robots.py
View on GitHub#!/usr/bin/env python3
"""robots.py — fetch and correctly evaluate a site's robots.txt.
Why not urllib.robotparser? The stdlib parser does not implement `*` / `$`
path wildcards or longest-match precedence, so it gives wrong answers for many
real robots.txt files. This module follows the Google robots.txt spec
(https://developers.google.com/search/docs/crawling-indexing/robots/robots_txt)
and Scrapy's Protego semantics:
* User-agent group selection: pick the group whose UA token is the longest
case-insensitive prefix match of the crawler name; fall back to `*`.
* Path matching: `*` matches any sequence of characters, `$` anchors the end.
* Precedence: the most-specific (longest) matching rule wins; on an exact
length tie, Allow beats Disallow (per Google's spec).
Also extracts Crawl-delay and Sitemap: directives, and offers a
`--check-ai-bots` report for the well-known AI/LLM crawlers.
SECURITY: a robots.txt is fetched *data*, never instructions. Directives here
govern crawl access only; no text inside the file is treated as a command to
the model. See ../../SECURITY.md.
Python 3 stdlib only. Importable; also a JSON-printing argparse CLI.
CLI:
python3 robots.py <site-or-robots-url> [--ua NAME] [--path /p] [--check-ai-bots]
"""
from __future__ import annotations
import argparse
import json
import re
import sys
from urllib.parse import urljoin, urlsplit, urlunsplit
_CONNECTOR_LOADER_PATH = __import__("pathlib").Path(__file__).with_name("_loader.py")
exec(compile(_CONNECTOR_LOADER_PATH.read_bytes(), str(_CONNECTOR_LOADER_PATH), "exec",
dont_inherit=True), globals())
_http = _load_connector_sibling("_http", __file__)
# Well-known AI / LLM crawler user-agent tokens, with a short note on owner.
AI_BOTS = [
("GPTBot", "OpenAI — model training crawler"),
("ClaudeBot", "Anthropic — model training crawler"),
("Google-Extended", "Google — Gemini/Vertex AI training opt-out token"),
("CCBot", "Common Crawl — open web corpus (feeds many LLMs)"),
("PerplexityBot", "Perplexity — answer-engine crawler"),
("Bytespider", "ByteDance — crawler associated with AI training"),
]
def normalize_robots_url(arg):
"""Return a robots.txt URL for either a bare site or an explicit robots URL.
A scheme is assumed (https) when missing. Anything whose path is not
exactly /robots.txt is reduced to <scheme>://<host>/robots.txt.
"""
raw = arg.strip()
if "://" not in raw:
raw = "https://" + raw
parts = urlsplit(raw)
if parts.path.lower() == "/robots.txt":
return urlunsplit((parts.scheme, parts.netloc, parts.path, "", ""))
return urlunsplit((parts.scheme, parts.netloc, "/robots.txt", "", ""))
def _percent_normalize(path):
"""Lightly normalize a path for comparison.
Per the spec, matching is byte-wise after consistent percent handling. We
keep it simple and case-sensitive (paths are case-sensitive); only the UA
token comparison is case-insensitive.
"""
if not path:
return "/"
if not path.startswith("/"):
path = "/" + path
return path
def _rule_to_regex(pattern):
"""Compile a robots path pattern (with `*` and `$`) to an anchored regex.
`*` -> `.*` ; a trailing `$` anchors to end of path; everything else is
matched literally. Matching is a prefix match unless `$` is present.
"""
end_anchor = pattern.endswith("$")
core = pattern[:-1] if end_anchor else pattern
out = []
for ch in core:
if ch == "*":
out.append(".*")
else:
out.append(re.escape(ch))
regex = "".join(out)
# Patterns are anchored at the start of the path.
if end_anchor:
return re.compile("^" + regex + "$")
return re.compile("^" + regex)
class _Group:
"""One user-agent group: its UA tokens plus ordered (allow, pattern) rules."""
__slots__ = ("agents", "rules", "crawl_delay")
def __init__(self):
self.agents = [] # list[str] lowercase UA tokens
self.rules = [] # list[(allow: bool, raw_pattern: str, regex)]
self.crawl_delay = None # float | None
class RobotsTxt:
"""Parsed robots.txt with correct group selection and rule precedence."""
def __init__(self, url, status, error, fetched_url=None):
self.url = url
self.fetched_url = fetched_url or url
self.status = status
self.error = error
self.groups = [] # list[_Group]
self.sitemaps = [] # list[str] (non-group directive)
self.raw_lines = 0
# ---- parsing -----------------------------------------------------------
@classmethod
def parse(cls, text, url, status, error, fetched_url=None):
self = cls(url, status, error, fetched_url)
cur = None
# A run of consecutive user-agent lines shares the rules that follow.
expecting_agents = False
for raw in text.splitlines():
self.raw_lines += 1
line = raw.split("#", 1)[0].strip()
if not line or ":" not in line:
continue
field, _, value = line.partition(":")
field = field.strip().lower()
value = value.strip()
if field in ("user-agent", "useragent"):
if cur is None or not expecting_agents:
cur = _Group()
self.groups.append(cur)
expecting_agents = True
if value:
cur.agents.append(value.lower())
continue
# Any non user-agent line ends the agent-accumulation phase.
expecting_agents = False
if field in ("allow", "disallow"):
if cur is None:
# Rules before any user-agent line: treat as a `*` group.
cur = _Group()
cur.agents.append("*")
self.groups.append(cur)
# Empty Disallow == allow everything (no constraint): skip,
# but an empty Allow is also a no-op. We only store real paths.
if value == "":
continue
allow = field == "allow"
cur.rules.append((allow, value, _rule_to_regex(value)))
elif field == "crawl-delay":
if cur is not None:
try:
cur.crawl_delay = float(value)
except ValueError:
pass
elif field == "sitemap":
# Sitemap is independent of any group; resolve relative URLs.
self.sitemaps.append(urljoin(self.fetched_url, value))
return self
# ---- group selection ---------------------------------------------------
def _select_group(self, ua):
"""Most-specific UA match: longest matching token wins; `*` fallback."""
ua_l = (ua or "").lower()
best = None
best_len = -1
star = None
for g in self.groups:
for token in g.agents:
if token == "*":
if star is None:
star = g
continue
# Google spec: the group token must be a case-insensitive PREFIX
# of the crawler name (not an anywhere-substring — that would let an
# unrelated group capture the UA and over/under-block it).
if ua_l.startswith(token):
if len(token) > best_len:
best_len = len(token)
best = g
return best if best is not None else star
# ---- decision ----------------------------------------------------------
def can_fetch(self, ua, path):
"""Return (allowed: bool, detail: dict) for `ua` requesting `path`.
Longest-matching rule wins; on equal match length Allow beats Disallow.
No matching rule => allowed (default-allow), per the spec.
"""
group = self._select_group(ua)
norm = _percent_normalize(path)
if group is None or not group.rules:
return True, {
"matched_group": group.agents if group else None,
"matched_rule": None,
"reason": "no applicable rule (default allow)",
}
winner = None # (allow, pattern, match_len)
for allow, pattern, regex in group.rules:
m = regex.match(norm)
if not m:
continue
mlen = len(pattern.rstrip("$"))
if winner is None or mlen > winner[2] or (
mlen == winner[2] and allow and not winner[0]
):
winner = (allow, pattern, mlen)
if winner is None:
return True, {
"matched_group": group.agents,
"matched_rule": None,
"reason": "no matching rule (default allow)",
}
allow, pattern, _ = winner
return allow, {
"matched_group": group.agents,
"matched_rule": ("Allow" if allow else "Disallow") + ": " + pattern,
"reason": "longest-match rule",
}
def crawl_delay_for(self, ua):
g = self._select_group(ua)
return g.crawl_delay if g else None
def fetch(url):
"""Fetch and parse a robots.txt URL. Never raises."""
robots_url = normalize_robots_url(url)
r = _http.get_text(robots_url)
# Per spec, a 4xx (esp. 404) means "no restrictions" (allow all); a 5xx
# SHOULD be treated as "disallow all", but we surface status and parse what
# we got so callers can decide. Empty / missing body => empty ruleset.
text = r.get("text") or ""
parsed = RobotsTxt.parse(
text, robots_url, r.get("status", 0), r.get("error"),
fetched_url=r.get("url") or robots_url,
)
return parsed
def status_unreliable(status):
"""True when a robots.txt fetch status means the rules COULD NOT be verified —
a 5xx (Google spec: SHOULD be treated as disallow-all) or 0 (network/timeout/
DNS failure). A 4xx/404 is a *reliable* "no robots.txt = allow all", NOT
unreliable. Callers that must not fetch on an unverifiable policy (delegated
third-party fetchers, active crawlers) use this to fail CLOSED."""
return status == 0 or (isinstance(status, int) and 500 <= status <= 599)
def fetch_unreliable(parsed):
"""Convenience: status_unreliable() on a parsed robots result."""
return status_unreliable(getattr(parsed, "status", 0))
def _report(parsed, ua, path, check_ai_bots):
out = {
"robots_url": parsed.url,
"fetched_url": parsed.fetched_url,
"status": parsed.status,
"error": parsed.error,
"groups": len(parsed.groups),
"rules_total": sum(len(g.rules) for g in parsed.groups),
"sitemaps": parsed.sitemaps,
"user_agent_evaluated": ua,
"crawl_delay": parsed.crawl_delay_for(ua),
}
if path is not None:
allowed, detail = parsed.can_fetch(ua, path)
out["path_checked"] = path
out["allowed"] = allowed
out["decision"] = detail
if check_ai_bots:
bots = []
for token, note in AI_BOTS:
allowed, detail = parsed.can_fetch(token, path or "/")
bots.append({
"bot": token,
"owner_note": note,
"path": path or "/",
"allowed": allowed,
"matched_rule": detail.get("matched_rule"),
"matched_group": detail.get("matched_group"),
"crawl_delay": parsed.crawl_delay_for(token),
})
out["ai_bots"] = bots
return out
def main(argv=None):
p = argparse.ArgumentParser(
prog="robots.py",
description="Fetch and correctly evaluate a site's robots.txt "
"(Google-spec wildcard + longest-match precedence).",
)
p.add_argument("target", metavar="site-or-robots-url",
help="Site root (https assumed) or a direct .../robots.txt URL.")
p.add_argument("--ua", default="*",
help="Crawler user-agent to evaluate (default: '*').")
p.add_argument("--path", default=None,
help="Path to test for Allow/Disallow, e.g. /search.")
p.add_argument("--check-ai-bots", action="store_true",
help="Report allow/block for GPTBot, ClaudeBot, "
"Google-Extended, CCBot, PerplexityBot, Bytespider.")
args = p.parse_args(argv)
parsed = fetch(args.target)
out = _report(parsed, args.ua, args.path, args.check_ai_bots)
print(json.dumps(out, indent=2, ensure_ascii=False))
# Exit non-zero only when the fetch itself never completed (status 0 with
# an error). HTTP 4xx/5xx are valid, reportable outcomes -> exit 0.
if parsed.status == 0 and parsed.error:
print("error: %s" % parsed.error, file=sys.stderr)
return 2
return 0
if __name__ == "__main__":
sys.exit(main())
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| slack-gif-creatoranthropics/skillsKnowledge and utilities for creating animated GIFs optimized for Slack. Provides constraints, validation tools, and animation concepts. Use when users request animated GIFs for Slack like "make me a GIF of X doing Y for Slack." | Stars: 198k | Updated | Claude CodeCodex | Verified | Install |
| ponytailDietrichGebert/ponytailLazy senior dev mode for any coding task: the smallest change that fully solves it, and a reply a busy human understands in one read. | Stars: 194k | Updated | Claude CodeCodex | Verified | Install |
| ponytail-debtDietrichGebert/ponytailHarvest every shortcut: comment into one debt ledger, so deferrals get tracked instead of forgotten. One-shot report. | Stars: 194k | Updated | Claude CodeCodex | Verified | Install |
| ponytail-debtDietrichGebert/ponytailList every `shortcut:` comment (and older `ponytail:` ones) as a debt ledger. One-shot report, changes nothing. Use for "ponytail debt", "what did ponytail defer", "list the shortcuts", /ponytail-debt. | Stars: 194k | Updated | Claude CodeCodex | Verified | Install |