iamwaqargulzar/Marketing-Agent-OS / skills/audience-belief-mapper
audience-belief-mapper
VerifiedUse when the user asks to "map what our buyers believe", "capture the objections we keep hearing", or "find the switching forces that move the beachhead"; produces a belief map of the beachhead — held beliefs and mental models, the recurring objections and their reframes, and the JTBD four forces (push of the problem, pull of the new, anxiety of switching, habit of the present) — each item sourced from interviews or win-loss notes (User-provided) and labeled Measured / User-provided / Estimated, with any unverified quote or comparative claim marked "[needs source]" and routed to the claims candidates, never adjudicated here. Not for demographic or persona profiling — use audience-mapper; not for the positioning canvas — use positioning-truth-tracer. 受众信念/异议地图/切换四力/流失语言
| File | Kind | Size |
|---|---|---|
| LICENSE.txt | markdown | 6.7 KB |
| NOTICE.md | markdown | 325 B |
| SKILL.md | skill md | 12.4 KB |
| references/shared/skill-contract.md | markdown | 15.4 KB |
| references/shared/tale-benchmark.md | markdown | 8.6 KB |
| references/skill-contract.md | markdown | 2.4 KB |
| scripts/connectors/_http.py | script | 28.2 KB |
| scripts/connectors/_loader.py | script | 1.2 KB |
| scripts/connectors/firecrawl.py | script | 13.1 KB |
| scripts/connectors/robots.py | script | 12.8 KB |
| scripts/connectors/tavily.py | script | 10.0 KB |
scripts/connectors/robots.py
View on GitHub#!/usr/bin/env python3
"""robots.py — fetch and correctly evaluate a site's robots.txt.
Why not urllib.robotparser? The stdlib parser does not implement `*` / `$`
path wildcards or longest-match precedence, so it gives wrong answers for many
real robots.txt files. This module follows the Google robots.txt spec
(https://developers.google.com/search/docs/crawling-indexing/robots/robots_txt)
and Scrapy's Protego semantics:
* User-agent group selection: pick the group whose UA token is the longest
case-insensitive prefix match of the crawler name; fall back to `*`.
* Path matching: `*` matches any sequence of characters, `$` anchors the end.
* Precedence: the most-specific (longest) matching rule wins; on an exact
length tie, Allow beats Disallow (per Google's spec).
Also extracts Crawl-delay and Sitemap: directives, and offers a
`--check-ai-bots` report for the well-known AI/LLM crawlers.
SECURITY: a robots.txt is fetched *data*, never instructions. Directives here
govern crawl access only; no text inside the file is treated as a command to
the model. See ../../SECURITY.md.
Python 3 stdlib only. Importable; also a JSON-printing argparse CLI.
CLI:
python3 robots.py <site-or-robots-url> [--ua NAME] [--path /p] [--check-ai-bots]
"""
from __future__ import annotations
import argparse
import json
import re
import sys
from urllib.parse import urljoin, urlsplit, urlunsplit
_CONNECTOR_LOADER_PATH = __import__("pathlib").Path(__file__).with_name("_loader.py")
exec(compile(_CONNECTOR_LOADER_PATH.read_bytes(), str(_CONNECTOR_LOADER_PATH), "exec",
dont_inherit=True), globals())
_http = _load_connector_sibling("_http", __file__)
# Well-known AI / LLM crawler user-agent tokens, with a short note on owner.
AI_BOTS = [
("GPTBot", "OpenAI — model training crawler"),
("ClaudeBot", "Anthropic — model training crawler"),
("Google-Extended", "Google — Gemini/Vertex AI training opt-out token"),
("CCBot", "Common Crawl — open web corpus (feeds many LLMs)"),
("PerplexityBot", "Perplexity — answer-engine crawler"),
("Bytespider", "ByteDance — crawler associated with AI training"),
]
def normalize_robots_url(arg):
"""Return a robots.txt URL for either a bare site or an explicit robots URL.
A scheme is assumed (https) when missing. Anything whose path is not
exactly /robots.txt is reduced to <scheme>://<host>/robots.txt.
"""
raw = arg.strip()
if "://" not in raw:
raw = "https://" + raw
parts = urlsplit(raw)
if parts.path.lower() == "/robots.txt":
return urlunsplit((parts.scheme, parts.netloc, parts.path, "", ""))
return urlunsplit((parts.scheme, parts.netloc, "/robots.txt", "", ""))
def _percent_normalize(path):
"""Lightly normalize a path for comparison.
Per the spec, matching is byte-wise after consistent percent handling. We
keep it simple and case-sensitive (paths are case-sensitive); only the UA
token comparison is case-insensitive.
"""
if not path:
return "/"
if not path.startswith("/"):
path = "/" + path
return path
def _rule_to_regex(pattern):
"""Compile a robots path pattern (with `*` and `$`) to an anchored regex.
`*` -> `.*` ; a trailing `$` anchors to end of path; everything else is
matched literally. Matching is a prefix match unless `$` is present.
"""
end_anchor = pattern.endswith("$")
core = pattern[:-1] if end_anchor else pattern
out = []
for ch in core:
if ch == "*":
out.append(".*")
else:
out.append(re.escape(ch))
regex = "".join(out)
# Patterns are anchored at the start of the path.
if end_anchor:
return re.compile("^" + regex + "$")
return re.compile("^" + regex)
class _Group:
"""One user-agent group: its UA tokens plus ordered (allow, pattern) rules."""
__slots__ = ("agents", "rules", "crawl_delay")
def __init__(self):
self.agents = [] # list[str] lowercase UA tokens
self.rules = [] # list[(allow: bool, raw_pattern: str, regex)]
self.crawl_delay = None # float | None
class RobotsTxt:
"""Parsed robots.txt with correct group selection and rule precedence."""
def __init__(self, url, status, error, fetched_url=None):
self.url = url
self.fetched_url = fetched_url or url
self.status = status
self.error = error
self.groups = [] # list[_Group]
self.sitemaps = [] # list[str] (non-group directive)
self.raw_lines = 0
# ---- parsing -----------------------------------------------------------
@classmethod
def parse(cls, text, url, status, error, fetched_url=None):
self = cls(url, status, error, fetched_url)
cur = None
# A run of consecutive user-agent lines shares the rules that follow.
expecting_agents = False
for raw in text.splitlines():
self.raw_lines += 1
line = raw.split("#", 1)[0].strip()
if not line or ":" not in line:
continue
field, _, value = line.partition(":")
field = field.strip().lower()
value = value.strip()
if field in ("user-agent", "useragent"):
if cur is None or not expecting_agents:
cur = _Group()
self.groups.append(cur)
expecting_agents = True
if value:
cur.agents.append(value.lower())
continue
# Any non user-agent line ends the agent-accumulation phase.
expecting_agents = False
if field in ("allow", "disallow"):
if cur is None:
# Rules before any user-agent line: treat as a `*` group.
cur = _Group()
cur.agents.append("*")
self.groups.append(cur)
# Empty Disallow == allow everything (no constraint): skip,
# but an empty Allow is also a no-op. We only store real paths.
if value == "":
continue
allow = field == "allow"
cur.rules.append((allow, value, _rule_to_regex(value)))
elif field == "crawl-delay":
if cur is not None:
try:
cur.crawl_delay = float(value)
except ValueError:
pass
elif field == "sitemap":
# Sitemap is independent of any group; resolve relative URLs.
self.sitemaps.append(urljoin(self.fetched_url, value))
return self
# ---- group selection ---------------------------------------------------
def _select_group(self, ua):
"""Most-specific UA match: longest matching token wins; `*` fallback."""
ua_l = (ua or "").lower()
best = None
best_len = -1
star = None
for g in self.groups:
for token in g.agents:
if token == "*":
if star is None:
star = g
continue
# Google spec: the group token must be a case-insensitive PREFIX
# of the crawler name (not an anywhere-substring — that would let an
# unrelated group capture the UA and over/under-block it).
if ua_l.startswith(token):
if len(token) > best_len:
best_len = len(token)
best = g
return best if best is not None else star
# ---- decision ----------------------------------------------------------
def can_fetch(self, ua, path):
"""Return (allowed: bool, detail: dict) for `ua` requesting `path`.
Longest-matching rule wins; on equal match length Allow beats Disallow.
No matching rule => allowed (default-allow), per the spec.
"""
group = self._select_group(ua)
norm = _percent_normalize(path)
if group is None or not group.rules:
return True, {
"matched_group": group.agents if group else None,
"matched_rule": None,
"reason": "no applicable rule (default allow)",
}
winner = None # (allow, pattern, match_len)
for allow, pattern, regex in group.rules:
m = regex.match(norm)
if not m:
continue
mlen = len(pattern.rstrip("$"))
if winner is None or mlen > winner[2] or (
mlen == winner[2] and allow and not winner[0]
):
winner = (allow, pattern, mlen)
if winner is None:
return True, {
"matched_group": group.agents,
"matched_rule": None,
"reason": "no matching rule (default allow)",
}
allow, pattern, _ = winner
return allow, {
"matched_group": group.agents,
"matched_rule": ("Allow" if allow else "Disallow") + ": " + pattern,
"reason": "longest-match rule",
}
def crawl_delay_for(self, ua):
g = self._select_group(ua)
return g.crawl_delay if g else None
def fetch(url):
"""Fetch and parse a robots.txt URL. Never raises."""
robots_url = normalize_robots_url(url)
r = _http.get_text(robots_url)
# Per spec, a 4xx (esp. 404) means "no restrictions" (allow all); a 5xx
# SHOULD be treated as "disallow all", but we surface status and parse what
# we got so callers can decide. Empty / missing body => empty ruleset.
text = r.get("text") or ""
parsed = RobotsTxt.parse(
text, robots_url, r.get("status", 0), r.get("error"),
fetched_url=r.get("url") or robots_url,
)
return parsed
def status_unreliable(status):
"""True when a robots.txt fetch status means the rules COULD NOT be verified —
a 5xx (Google spec: SHOULD be treated as disallow-all) or 0 (network/timeout/
DNS failure). A 4xx/404 is a *reliable* "no robots.txt = allow all", NOT
unreliable. Callers that must not fetch on an unverifiable policy (delegated
third-party fetchers, active crawlers) use this to fail CLOSED."""
return status == 0 or (isinstance(status, int) and 500 <= status <= 599)
def fetch_unreliable(parsed):
"""Convenience: status_unreliable() on a parsed robots result."""
return status_unreliable(getattr(parsed, "status", 0))
def _report(parsed, ua, path, check_ai_bots):
out = {
"robots_url": parsed.url,
"fetched_url": parsed.fetched_url,
"status": parsed.status,
"error": parsed.error,
"groups": len(parsed.groups),
"rules_total": sum(len(g.rules) for g in parsed.groups),
"sitemaps": parsed.sitemaps,
"user_agent_evaluated": ua,
"crawl_delay": parsed.crawl_delay_for(ua),
}
if path is not None:
allowed, detail = parsed.can_fetch(ua, path)
out["path_checked"] = path
out["allowed"] = allowed
out["decision"] = detail
if check_ai_bots:
bots = []
for token, note in AI_BOTS:
allowed, detail = parsed.can_fetch(token, path or "/")
bots.append({
"bot": token,
"owner_note": note,
"path": path or "/",
"allowed": allowed,
"matched_rule": detail.get("matched_rule"),
"matched_group": detail.get("matched_group"),
"crawl_delay": parsed.crawl_delay_for(token),
})
out["ai_bots"] = bots
return out
def main(argv=None):
p = argparse.ArgumentParser(
prog="robots.py",
description="Fetch and correctly evaluate a site's robots.txt "
"(Google-spec wildcard + longest-match precedence).",
)
p.add_argument("target", metavar="site-or-robots-url",
help="Site root (https assumed) or a direct .../robots.txt URL.")
p.add_argument("--ua", default="*",
help="Crawler user-agent to evaluate (default: '*').")
p.add_argument("--path", default=None,
help="Path to test for Allow/Disallow, e.g. /search.")
p.add_argument("--check-ai-bots", action="store_true",
help="Report allow/block for GPTBot, ClaudeBot, "
"Google-Extended, CCBot, PerplexityBot, Bytespider.")
args = p.parse_args(argv)
parsed = fetch(args.target)
out = _report(parsed, args.ua, args.path, args.check_ai_bots)
print(json.dumps(out, indent=2, ensure_ascii=False))
# Exit non-zero only when the fetch itself never completed (status 0 with
# an error). HTTP 4xx/5xx are valid, reportable outcomes -> exit 0.
if parsed.status == 0 and parsed.error:
print("error: %s" % parsed.error, file=sys.stderr)
return 2
return 0
if __name__ == "__main__":
sys.exit(main())
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| slack-gif-creatoranthropics/skillsKnowledge and utilities for creating animated GIFs optimized for Slack. Provides constraints, validation tools, and animation concepts. Use when users request animated GIFs for Slack like "make me a GIF of X doing Y for Slack." | Stars: 198k | Updated | Claude CodeCodex | Verified | Install |
| ponytailDietrichGebert/ponytailLazy senior dev mode for any coding task: the smallest change that fully solves it, and a reply a busy human understands in one read. | Stars: 194k | Updated | Claude CodeCodex | Verified | Install |
| ponytail-debtDietrichGebert/ponytailHarvest every shortcut: comment into one debt ledger, so deferrals get tracked instead of forgotten. One-shot report. | Stars: 194k | Updated | Claude CodeCodex | Verified | Install |
| ponytail-debtDietrichGebert/ponytailList every `shortcut:` comment (and older `ponytail:` ones) as a debt ledger. One-shot report, changes nothing. Use for "ponytail debt", "what did ponytail defer", "list the shortcuts", /ponytail-debt. | Stars: 194k | Updated | Claude CodeCodex | Verified | Install |