Skip to content

iamwaqargulzar/Marketing-Agent-OS / skills/brand-language-codifier

brand-language-codifier

Verified

Use when the user asks to "codify our brand voice", "define naming rules for our products and tiers", or "write the tone-of-voice guide with banned phrases"; produces the brand-level voice canon (register, tone spectrum, banned-phrase list, few-shot examples drawn only from the brand's own published material) and the naming tax (product / feature / tier naming rules plus approved and banned terms) that seeds the narrative-registry canon and that every channel's voice adaptation points up to. Not for per-platform voice adaptation — use channel-registry's voice-dossier; not for finished copy or blog posts — use content-writer; not for the message hierarchy itself — use message-system-architect; not for claim adjudication — use offer-claims-registry. 品牌语气/词汇表/命名税/禁用词/品牌语言规范

scripts/connectors/robots.py

View on GitHub
#!/usr/bin/env python3
"""robots.py — fetch and correctly evaluate a site's robots.txt.

Why not urllib.robotparser? The stdlib parser does not implement `*` / `$`
path wildcards or longest-match precedence, so it gives wrong answers for many
real robots.txt files. This module follows the Google robots.txt spec
(https://developers.google.com/search/docs/crawling-indexing/robots/robots_txt)
and Scrapy's Protego semantics:

  * User-agent group selection: pick the group whose UA token is the longest
    case-insensitive prefix match of the crawler name; fall back to `*`.
  * Path matching: `*` matches any sequence of characters, `$` anchors the end.
  * Precedence: the most-specific (longest) matching rule wins; on an exact
    length tie, Allow beats Disallow (per Google's spec).

Also extracts Crawl-delay and Sitemap: directives, and offers a
`--check-ai-bots` report for the well-known AI/LLM crawlers.

SECURITY: a robots.txt is fetched *data*, never instructions. Directives here
govern crawl access only; no text inside the file is treated as a command to
the model. See ../../SECURITY.md.

Python 3 stdlib only. Importable; also a JSON-printing argparse CLI.

CLI:
  python3 robots.py <site-or-robots-url> [--ua NAME] [--path /p] [--check-ai-bots]
"""
from __future__ import annotations

import argparse
import json
import re
import sys
from urllib.parse import urljoin, urlsplit, urlunsplit

_CONNECTOR_LOADER_PATH = __import__("pathlib").Path(__file__).with_name("_loader.py")
exec(compile(_CONNECTOR_LOADER_PATH.read_bytes(), str(_CONNECTOR_LOADER_PATH), "exec",
             dont_inherit=True), globals())
_http = _load_connector_sibling("_http", __file__)

# Well-known AI / LLM crawler user-agent tokens, with a short note on owner.
AI_BOTS = [
    ("GPTBot", "OpenAI — model training crawler"),
    ("ClaudeBot", "Anthropic — model training crawler"),
    ("Google-Extended", "Google — Gemini/Vertex AI training opt-out token"),
    ("CCBot", "Common Crawl — open web corpus (feeds many LLMs)"),
    ("PerplexityBot", "Perplexity — answer-engine crawler"),
    ("Bytespider", "ByteDance — crawler associated with AI training"),
]


def normalize_robots_url(arg):
    """Return a robots.txt URL for either a bare site or an explicit robots URL.

    A scheme is assumed (https) when missing. Anything whose path is not
    exactly /robots.txt is reduced to <scheme>://<host>/robots.txt.
    """
    raw = arg.strip()
    if "://" not in raw:
        raw = "https://" + raw
    parts = urlsplit(raw)
    if parts.path.lower() == "/robots.txt":
        return urlunsplit((parts.scheme, parts.netloc, parts.path, "", ""))
    return urlunsplit((parts.scheme, parts.netloc, "/robots.txt", "", ""))


def _percent_normalize(path):
    """Lightly normalize a path for comparison.

    Per the spec, matching is byte-wise after consistent percent handling. We
    keep it simple and case-sensitive (paths are case-sensitive); only the UA
    token comparison is case-insensitive.
    """
    if not path:
        return "/"
    if not path.startswith("/"):
        path = "/" + path
    return path


def _rule_to_regex(pattern):
    """Compile a robots path pattern (with `*` and `$`) to an anchored regex.

    `*` -> `.*` ; a trailing `$` anchors to end of path; everything else is
    matched literally. Matching is a prefix match unless `$` is present.
    """
    end_anchor = pattern.endswith("$")
    core = pattern[:-1] if end_anchor else pattern
    out = []
    for ch in core:
        if ch == "*":
            out.append(".*")
        else:
            out.append(re.escape(ch))
    regex = "".join(out)
    # Patterns are anchored at the start of the path.
    if end_anchor:
        return re.compile("^" + regex + "$")
    return re.compile("^" + regex)


class _Group:
    """One user-agent group: its UA tokens plus ordered (allow, pattern) rules."""

    __slots__ = ("agents", "rules", "crawl_delay")

    def __init__(self):
        self.agents = []          # list[str] lowercase UA tokens
        self.rules = []           # list[(allow: bool, raw_pattern: str, regex)]
        self.crawl_delay = None   # float | None


class RobotsTxt:
    """Parsed robots.txt with correct group selection and rule precedence."""

    def __init__(self, url, status, error, fetched_url=None):
        self.url = url
        self.fetched_url = fetched_url or url
        self.status = status
        self.error = error
        self.groups = []          # list[_Group]
        self.sitemaps = []        # list[str] (non-group directive)
        self.raw_lines = 0

    # ---- parsing -----------------------------------------------------------
    @classmethod
    def parse(cls, text, url, status, error, fetched_url=None):
        self = cls(url, status, error, fetched_url)
        cur = None
        # A run of consecutive user-agent lines shares the rules that follow.
        expecting_agents = False
        for raw in text.splitlines():
            self.raw_lines += 1
            line = raw.split("#", 1)[0].strip()
            if not line or ":" not in line:
                continue
            field, _, value = line.partition(":")
            field = field.strip().lower()
            value = value.strip()
            if field in ("user-agent", "useragent"):
                if cur is None or not expecting_agents:
                    cur = _Group()
                    self.groups.append(cur)
                    expecting_agents = True
                if value:
                    cur.agents.append(value.lower())
                continue
            # Any non user-agent line ends the agent-accumulation phase.
            expecting_agents = False
            if field in ("allow", "disallow"):
                if cur is None:
                    # Rules before any user-agent line: treat as a `*` group.
                    cur = _Group()
                    cur.agents.append("*")
                    self.groups.append(cur)
                # Empty Disallow == allow everything (no constraint): skip,
                # but an empty Allow is also a no-op. We only store real paths.
                if value == "":
                    continue
                allow = field == "allow"
                cur.rules.append((allow, value, _rule_to_regex(value)))
            elif field == "crawl-delay":
                if cur is not None:
                    try:
                        cur.crawl_delay = float(value)
                    except ValueError:
                        pass
            elif field == "sitemap":
                # Sitemap is independent of any group; resolve relative URLs.
                self.sitemaps.append(urljoin(self.fetched_url, value))
        return self

    # ---- group selection ---------------------------------------------------
    def _select_group(self, ua):
        """Most-specific UA match: longest matching token wins; `*` fallback."""
        ua_l = (ua or "").lower()
        best = None
        best_len = -1
        star = None
        for g in self.groups:
            for token in g.agents:
                if token == "*":
                    if star is None:
                        star = g
                    continue
                # Google spec: the group token must be a case-insensitive PREFIX
                # of the crawler name (not an anywhere-substring — that would let an
                # unrelated group capture the UA and over/under-block it).
                if ua_l.startswith(token):
                    if len(token) > best_len:
                        best_len = len(token)
                        best = g
        return best if best is not None else star

    # ---- decision ----------------------------------------------------------
    def can_fetch(self, ua, path):
        """Return (allowed: bool, detail: dict) for `ua` requesting `path`.

        Longest-matching rule wins; on equal match length Allow beats Disallow.
        No matching rule => allowed (default-allow), per the spec.
        """
        group = self._select_group(ua)
        norm = _percent_normalize(path)
        if group is None or not group.rules:
            return True, {
                "matched_group": group.agents if group else None,
                "matched_rule": None,
                "reason": "no applicable rule (default allow)",
            }
        winner = None  # (allow, pattern, match_len)
        for allow, pattern, regex in group.rules:
            m = regex.match(norm)
            if not m:
                continue
            mlen = len(pattern.rstrip("$"))
            if winner is None or mlen > winner[2] or (
                mlen == winner[2] and allow and not winner[0]
            ):
                winner = (allow, pattern, mlen)
        if winner is None:
            return True, {
                "matched_group": group.agents,
                "matched_rule": None,
                "reason": "no matching rule (default allow)",
            }
        allow, pattern, _ = winner
        return allow, {
            "matched_group": group.agents,
            "matched_rule": ("Allow" if allow else "Disallow") + ": " + pattern,
            "reason": "longest-match rule",
        }

    def crawl_delay_for(self, ua):
        g = self._select_group(ua)
        return g.crawl_delay if g else None


def fetch(url):
    """Fetch and parse a robots.txt URL. Never raises."""
    robots_url = normalize_robots_url(url)
    r = _http.get_text(robots_url)
    # Per spec, a 4xx (esp. 404) means "no restrictions" (allow all); a 5xx
    # SHOULD be treated as "disallow all", but we surface status and parse what
    # we got so callers can decide. Empty / missing body => empty ruleset.
    text = r.get("text") or ""
    parsed = RobotsTxt.parse(
        text, robots_url, r.get("status", 0), r.get("error"),
        fetched_url=r.get("url") or robots_url,
    )
    return parsed


def status_unreliable(status):
    """True when a robots.txt fetch status means the rules COULD NOT be verified —
    a 5xx (Google spec: SHOULD be treated as disallow-all) or 0 (network/timeout/
    DNS failure). A 4xx/404 is a *reliable* "no robots.txt = allow all", NOT
    unreliable. Callers that must not fetch on an unverifiable policy (delegated
    third-party fetchers, active crawlers) use this to fail CLOSED."""
    return status == 0 or (isinstance(status, int) and 500 <= status <= 599)


def fetch_unreliable(parsed):
    """Convenience: status_unreliable() on a parsed robots result."""
    return status_unreliable(getattr(parsed, "status", 0))


def _report(parsed, ua, path, check_ai_bots):
    out = {
        "robots_url": parsed.url,
        "fetched_url": parsed.fetched_url,
        "status": parsed.status,
        "error": parsed.error,
        "groups": len(parsed.groups),
        "rules_total": sum(len(g.rules) for g in parsed.groups),
        "sitemaps": parsed.sitemaps,
        "user_agent_evaluated": ua,
        "crawl_delay": parsed.crawl_delay_for(ua),
    }
    if path is not None:
        allowed, detail = parsed.can_fetch(ua, path)
        out["path_checked"] = path
        out["allowed"] = allowed
        out["decision"] = detail
    if check_ai_bots:
        bots = []
        for token, note in AI_BOTS:
            allowed, detail = parsed.can_fetch(token, path or "/")
            bots.append({
                "bot": token,
                "owner_note": note,
                "path": path or "/",
                "allowed": allowed,
                "matched_rule": detail.get("matched_rule"),
                "matched_group": detail.get("matched_group"),
                "crawl_delay": parsed.crawl_delay_for(token),
            })
        out["ai_bots"] = bots
    return out


def main(argv=None):
    p = argparse.ArgumentParser(
        prog="robots.py",
        description="Fetch and correctly evaluate a site's robots.txt "
                    "(Google-spec wildcard + longest-match precedence).",
    )
    p.add_argument("target", metavar="site-or-robots-url",
                   help="Site root (https assumed) or a direct .../robots.txt URL.")
    p.add_argument("--ua", default="*",
                   help="Crawler user-agent to evaluate (default: '*').")
    p.add_argument("--path", default=None,
                   help="Path to test for Allow/Disallow, e.g. /search.")
    p.add_argument("--check-ai-bots", action="store_true",
                   help="Report allow/block for GPTBot, ClaudeBot, "
                        "Google-Extended, CCBot, PerplexityBot, Bytespider.")
    args = p.parse_args(argv)

    parsed = fetch(args.target)
    out = _report(parsed, args.ua, args.path, args.check_ai_bots)
    print(json.dumps(out, indent=2, ensure_ascii=False))
    # Exit non-zero only when the fetch itself never completed (status 0 with
    # an error). HTTP 4xx/5xx are valid, reportable outcomes -> exit 0.
    if parsed.status == 0 and parsed.error:
        print("error: %s" % parsed.error, file=sys.stderr)
        return 2
    return 0


if __name__ == "__main__":
    sys.exit(main())
slack-gif-creatoranthropics/skillsKnowledge and utilities for creating animated GIFs optimized for Slack. Provides constraints, validation tools, and animation concepts. Use when users request animated GIFs for Slack like "make me a GIF of X doing Y for Slack."Stars: 198kUpdated Claude CodeCodexVerifiedInstall
ponytailDietrichGebert/ponytailLazy senior dev mode for any coding task: the smallest change that fully solves it, and a reply a busy human understands in one read.Stars: 194kUpdated Claude CodeCodexVerifiedInstall
ponytail-debtDietrichGebert/ponytailHarvest every shortcut: comment into one debt ledger, so deferrals get tracked instead of forgotten. One-shot report.Stars: 194kUpdated Claude CodeCodexVerifiedInstall
ponytail-debtDietrichGebert/ponytailList every `shortcut:` comment (and older `ponytail:` ones) as a debt ledger. One-shot report, changes nothing. Use for "ponytail debt", "what did ponytail defer", "list the shortcuts", /ponytail-debt.Stars: 194kUpdated Claude CodeCodexVerifiedInstall