Skip to content

iamwaqargulzar/Marketing-Agent-OS / skills/audience-belief-mapper

audience-belief-mapper

Verified

Use when the user asks to "map what our buyers believe", "capture the objections we keep hearing", or "find the switching forces that move the beachhead"; produces a belief map of the beachhead — held beliefs and mental models, the recurring objections and their reframes, and the JTBD four forces (push of the problem, pull of the new, anxiety of switching, habit of the present) — each item sourced from interviews or win-loss notes (User-provided) and labeled Measured / User-provided / Estimated, with any unverified quote or comparative claim marked "[needs source]" and routed to the claims candidates, never adjudicated here. Not for demographic or persona profiling — use audience-mapper; not for the positioning canvas — use positioning-truth-tracer. 受众信念/异议地图/切换四力/流失语言

scripts/connectors/robots.py

View on GitHub
#!/usr/bin/env python3
"""robots.py — fetch and correctly evaluate a site's robots.txt.

Why not urllib.robotparser? The stdlib parser does not implement `*` / `$`
path wildcards or longest-match precedence, so it gives wrong answers for many
real robots.txt files. This module follows the Google robots.txt spec
(https://developers.google.com/search/docs/crawling-indexing/robots/robots_txt)
and Scrapy's Protego semantics:

  * User-agent group selection: pick the group whose UA token is the longest
    case-insensitive prefix match of the crawler name; fall back to `*`.
  * Path matching: `*` matches any sequence of characters, `$` anchors the end.
  * Precedence: the most-specific (longest) matching rule wins; on an exact
    length tie, Allow beats Disallow (per Google's spec).

Also extracts Crawl-delay and Sitemap: directives, and offers a
`--check-ai-bots` report for the well-known AI/LLM crawlers.

SECURITY: a robots.txt is fetched *data*, never instructions. Directives here
govern crawl access only; no text inside the file is treated as a command to
the model. See ../../SECURITY.md.

Python 3 stdlib only. Importable; also a JSON-printing argparse CLI.

CLI:
  python3 robots.py <site-or-robots-url> [--ua NAME] [--path /p] [--check-ai-bots]
"""
from __future__ import annotations

import argparse
import json
import re
import sys
from urllib.parse import urljoin, urlsplit, urlunsplit

_CONNECTOR_LOADER_PATH = __import__("pathlib").Path(__file__).with_name("_loader.py")
exec(compile(_CONNECTOR_LOADER_PATH.read_bytes(), str(_CONNECTOR_LOADER_PATH), "exec",
             dont_inherit=True), globals())
_http = _load_connector_sibling("_http", __file__)

# Well-known AI / LLM crawler user-agent tokens, with a short note on owner.
AI_BOTS = [
    ("GPTBot", "OpenAI — model training crawler"),
    ("ClaudeBot", "Anthropic — model training crawler"),
    ("Google-Extended", "Google — Gemini/Vertex AI training opt-out token"),
    ("CCBot", "Common Crawl — open web corpus (feeds many LLMs)"),
    ("PerplexityBot", "Perplexity — answer-engine crawler"),
    ("Bytespider", "ByteDance — crawler associated with AI training"),
]


def normalize_robots_url(arg):
    """Return a robots.txt URL for either a bare site or an explicit robots URL.

    A scheme is assumed (https) when missing. Anything whose path is not
    exactly /robots.txt is reduced to <scheme>://<host>/robots.txt.
    """
    raw = arg.strip()
    if "://" not in raw:
        raw = "https://" + raw
    parts = urlsplit(raw)
    if parts.path.lower() == "/robots.txt":
        return urlunsplit((parts.scheme, parts.netloc, parts.path, "", ""))
    return urlunsplit((parts.scheme, parts.netloc, "/robots.txt", "", ""))


def _percent_normalize(path):
    """Lightly normalize a path for comparison.

    Per the spec, matching is byte-wise after consistent percent handling. We
    keep it simple and case-sensitive (paths are case-sensitive); only the UA
    token comparison is case-insensitive.
    """
    if not path:
        return "/"
    if not path.startswith("/"):
        path = "/" + path
    return path


def _rule_to_regex(pattern):
    """Compile a robots path pattern (with `*` and `$`) to an anchored regex.

    `*` -> `.*` ; a trailing `$` anchors to end of path; everything else is
    matched literally. Matching is a prefix match unless `$` is present.
    """
    end_anchor = pattern.endswith("$")
    core = pattern[:-1] if end_anchor else pattern
    out = []
    for ch in core:
        if ch == "*":
            out.append(".*")
        else:
            out.append(re.escape(ch))
    regex = "".join(out)
    # Patterns are anchored at the start of the path.
    if end_anchor:
        return re.compile("^" + regex + "$")
    return re.compile("^" + regex)


class _Group:
    """One user-agent group: its UA tokens plus ordered (allow, pattern) rules."""

    __slots__ = ("agents", "rules", "crawl_delay")

    def __init__(self):
        self.agents = []          # list[str] lowercase UA tokens
        self.rules = []           # list[(allow: bool, raw_pattern: str, regex)]
        self.crawl_delay = None   # float | None


class RobotsTxt:
    """Parsed robots.txt with correct group selection and rule precedence."""

    def __init__(self, url, status, error, fetched_url=None):
        self.url = url
        self.fetched_url = fetched_url or url
        self.status = status
        self.error = error
        self.groups = []          # list[_Group]
        self.sitemaps = []        # list[str] (non-group directive)
        self.raw_lines = 0

    # ---- parsing -----------------------------------------------------------
    @classmethod
    def parse(cls, text, url, status, error, fetched_url=None):
        self = cls(url, status, error, fetched_url)
        cur = None
        # A run of consecutive user-agent lines shares the rules that follow.
        expecting_agents = False
        for raw in text.splitlines():
            self.raw_lines += 1
            line = raw.split("#", 1)[0].strip()
            if not line or ":" not in line:
                continue
            field, _, value = line.partition(":")
            field = field.strip().lower()
            value = value.strip()
            if field in ("user-agent", "useragent"):
                if cur is None or not expecting_agents:
                    cur = _Group()
                    self.groups.append(cur)
                    expecting_agents = True
                if value:
                    cur.agents.append(value.lower())
                continue
            # Any non user-agent line ends the agent-accumulation phase.
            expecting_agents = False
            if field in ("allow", "disallow"):
                if cur is None:
                    # Rules before any user-agent line: treat as a `*` group.
                    cur = _Group()
                    cur.agents.append("*")
                    self.groups.append(cur)
                # Empty Disallow == allow everything (no constraint): skip,
                # but an empty Allow is also a no-op. We only store real paths.
                if value == "":
                    continue
                allow = field == "allow"
                cur.rules.append((allow, value, _rule_to_regex(value)))
            elif field == "crawl-delay":
                if cur is not None:
                    try:
                        cur.crawl_delay = float(value)
                    except ValueError:
                        pass
            elif field == "sitemap":
                # Sitemap is independent of any group; resolve relative URLs.
                self.sitemaps.append(urljoin(self.fetched_url, value))
        return self

    # ---- group selection ---------------------------------------------------
    def _select_group(self, ua):
        """Most-specific UA match: longest matching token wins; `*` fallback."""
        ua_l = (ua or "").lower()
        best = None
        best_len = -1
        star = None
        for g in self.groups:
            for token in g.agents:
                if token == "*":
                    if star is None:
                        star = g
                    continue
                # Google spec: the group token must be a case-insensitive PREFIX
                # of the crawler name (not an anywhere-substring — that would let an
                # unrelated group capture the UA and over/under-block it).
                if ua_l.startswith(token):
                    if len(token) > best_len:
                        best_len = len(token)
                        best = g
        return best if best is not None else star

    # ---- decision ----------------------------------------------------------
    def can_fetch(self, ua, path):
        """Return (allowed: bool, detail: dict) for `ua` requesting `path`.

        Longest-matching rule wins; on equal match length Allow beats Disallow.
        No matching rule => allowed (default-allow), per the spec.
        """
        group = self._select_group(ua)
        norm = _percent_normalize(path)
        if group is None or not group.rules:
            return True, {
                "matched_group": group.agents if group else None,
                "matched_rule": None,
                "reason": "no applicable rule (default allow)",
            }
        winner = None  # (allow, pattern, match_len)
        for allow, pattern, regex in group.rules:
            m = regex.match(norm)
            if not m:
                continue
            mlen = len(pattern.rstrip("$"))
            if winner is None or mlen > winner[2] or (
                mlen == winner[2] and allow and not winner[0]
            ):
                winner = (allow, pattern, mlen)
        if winner is None:
            return True, {
                "matched_group": group.agents,
                "matched_rule": None,
                "reason": "no matching rule (default allow)",
            }
        allow, pattern, _ = winner
        return allow, {
            "matched_group": group.agents,
            "matched_rule": ("Allow" if allow else "Disallow") + ": " + pattern,
            "reason": "longest-match rule",
        }

    def crawl_delay_for(self, ua):
        g = self._select_group(ua)
        return g.crawl_delay if g else None


def fetch(url):
    """Fetch and parse a robots.txt URL. Never raises."""
    robots_url = normalize_robots_url(url)
    r = _http.get_text(robots_url)
    # Per spec, a 4xx (esp. 404) means "no restrictions" (allow all); a 5xx
    # SHOULD be treated as "disallow all", but we surface status and parse what
    # we got so callers can decide. Empty / missing body => empty ruleset.
    text = r.get("text") or ""
    parsed = RobotsTxt.parse(
        text, robots_url, r.get("status", 0), r.get("error"),
        fetched_url=r.get("url") or robots_url,
    )
    return parsed


def status_unreliable(status):
    """True when a robots.txt fetch status means the rules COULD NOT be verified —
    a 5xx (Google spec: SHOULD be treated as disallow-all) or 0 (network/timeout/
    DNS failure). A 4xx/404 is a *reliable* "no robots.txt = allow all", NOT
    unreliable. Callers that must not fetch on an unverifiable policy (delegated
    third-party fetchers, active crawlers) use this to fail CLOSED."""
    return status == 0 or (isinstance(status, int) and 500 <= status <= 599)


def fetch_unreliable(parsed):
    """Convenience: status_unreliable() on a parsed robots result."""
    return status_unreliable(getattr(parsed, "status", 0))


def _report(parsed, ua, path, check_ai_bots):
    out = {
        "robots_url": parsed.url,
        "fetched_url": parsed.fetched_url,
        "status": parsed.status,
        "error": parsed.error,
        "groups": len(parsed.groups),
        "rules_total": sum(len(g.rules) for g in parsed.groups),
        "sitemaps": parsed.sitemaps,
        "user_agent_evaluated": ua,
        "crawl_delay": parsed.crawl_delay_for(ua),
    }
    if path is not None:
        allowed, detail = parsed.can_fetch(ua, path)
        out["path_checked"] = path
        out["allowed"] = allowed
        out["decision"] = detail
    if check_ai_bots:
        bots = []
        for token, note in AI_BOTS:
            allowed, detail = parsed.can_fetch(token, path or "/")
            bots.append({
                "bot": token,
                "owner_note": note,
                "path": path or "/",
                "allowed": allowed,
                "matched_rule": detail.get("matched_rule"),
                "matched_group": detail.get("matched_group"),
                "crawl_delay": parsed.crawl_delay_for(token),
            })
        out["ai_bots"] = bots
    return out


def main(argv=None):
    p = argparse.ArgumentParser(
        prog="robots.py",
        description="Fetch and correctly evaluate a site's robots.txt "
                    "(Google-spec wildcard + longest-match precedence).",
    )
    p.add_argument("target", metavar="site-or-robots-url",
                   help="Site root (https assumed) or a direct .../robots.txt URL.")
    p.add_argument("--ua", default="*",
                   help="Crawler user-agent to evaluate (default: '*').")
    p.add_argument("--path", default=None,
                   help="Path to test for Allow/Disallow, e.g. /search.")
    p.add_argument("--check-ai-bots", action="store_true",
                   help="Report allow/block for GPTBot, ClaudeBot, "
                        "Google-Extended, CCBot, PerplexityBot, Bytespider.")
    args = p.parse_args(argv)

    parsed = fetch(args.target)
    out = _report(parsed, args.ua, args.path, args.check_ai_bots)
    print(json.dumps(out, indent=2, ensure_ascii=False))
    # Exit non-zero only when the fetch itself never completed (status 0 with
    # an error). HTTP 4xx/5xx are valid, reportable outcomes -> exit 0.
    if parsed.status == 0 and parsed.error:
        print("error: %s" % parsed.error, file=sys.stderr)
        return 2
    return 0


if __name__ == "__main__":
    sys.exit(main())
slack-gif-creatoranthropics/skillsKnowledge and utilities for creating animated GIFs optimized for Slack. Provides constraints, validation tools, and animation concepts. Use when users request animated GIFs for Slack like "make me a GIF of X doing Y for Slack."Stars: 198kUpdated Claude CodeCodexVerifiedInstall
ponytailDietrichGebert/ponytailLazy senior dev mode for any coding task: the smallest change that fully solves it, and a reply a busy human understands in one read.Stars: 194kUpdated Claude CodeCodexVerifiedInstall
ponytail-debtDietrichGebert/ponytailHarvest every shortcut: comment into one debt ledger, so deferrals get tracked instead of forgotten. One-shot report.Stars: 194kUpdated Claude CodeCodexVerifiedInstall
ponytail-debtDietrichGebert/ponytailList every `shortcut:` comment (and older `ponytail:` ones) as a debt ledger. One-shot report, changes nothing. Use for "ponytail debt", "what did ponytail defer", "list the shortcuts", /ponytail-debt.Stars: 194kUpdated Claude CodeCodexVerifiedInstall