Skip to content

Liberty91LTD/cti-skills / skills/source-provenance

source-provenance

Verified

Use when you need to know where reporting actually came from: an article URL with "where does this come from?", "who is the original source?", "is this a rewrite?", or another skill needs the provenance chain before grading or citing. Follows links and attributions back to the originating vendor, government, victim or actor document and returns the chain, the primary's access and confidence language, dates and unresolved hops. Deterministic, no API key. Invoked by /quality-of-information-check, /threat-actor-profiling, /campaign-tracking and /intelligence-writing.

FileKindSize
SKILL.mdskill md12.2 KB
references/attribution-phrases.mdmarkdown8.0 KB
references/outlet-behaviour.mdmarkdown3.5 KB
references/source-types.mdmarkdown35.0 KB
scripts/resolve_provenance.pyexecutablescript61.7 KB

scripts/resolve_provenance.py

View on GitHub
#!/usr/bin/env python3
"""
resolve_provenance.py — resolve an article's provenance chain to its originating source.

Deterministic. Standard library only. No model in the loop: the chain this script
returns is data, and every field it could not determine is null with a reason.

Usage:
  resolve_provenance.py resolve <url> [<url> ...] [--max-hops 1|2] [--no-cache]
                                [--cache-dir DIR] [--fixtures MAP.json] [--today YYYY-MM-DD]
  resolve_provenance.py classify <url> [<url> ...]
  resolve_provenance.py text <url> [--no-cache] [--cache-dir DIR] [--fixtures MAP.json]

  resolve   follow links and named attributions to the primary; emit JSON on stdout
  classify  look a URL up in references/source-types.md; no network
  text      fetch a page, extract the article body to the local cache, print the
            path to the text file (used by /claim-extraction; the text is never
            included in any JSON output)

Exit codes:
  0  every input resolved to a primary (provenance_basis: script-resolved)
  1  usage or internal error
  2  at least one input did not resolve to a primary
  3  refused: more than 50 URLs or more than 2 hops requested

Fetcher policy (fixed; there is no override flag for robots.txt):
  - User agent: cti-skills-provenance/2.0 (+https://github.com/Liberty91LTD/cti-skills)
  - robots.txt honoured. A blocked page is reported; paste the text to continue.
  - One request in flight per host, 2 seconds between requests to the same host,
    at most 4 hosts in flight.
  - 429/503: honour Retry-After, else exponential backoff from 10s; three attempts,
    then the host is skipped for the rest of the run.
  - Local cache keyed by URL, revalidated with ETag / Last-Modified.
    TTL 7 days for press, 30 days for primaries. Never uploaded, never committed.
  - Hard cap: 50 URLs and 2 hops per invocation.
  - Timeouts 10s connect, 20s read. 5 MB size limit. PDFs are not parsed here.
  - Output contains extracted fields and anchor sentences only. Full text stays
    in the local cache.
"""

import argparse
import datetime as dt
import hashlib
import html
import http.client
import json
import os
import re
import sys
import threading
import time
import urllib.error
import urllib.parse
import urllib.request
import urllib.robotparser
from concurrent.futures import ThreadPoolExecutor
from html.parser import HTMLParser
from pathlib import Path

USER_AGENT = "cti-skills-provenance/2.0 (+https://github.com/Liberty91LTD/cti-skills)"
ROBOTS_AGENT = "cti-skills-provenance"
CONNECT_TIMEOUT = 10
READ_TIMEOUT = 20
MAX_BYTES = 5 * 1024 * 1024
HOST_SPACING = 2.0
MAX_HOSTS_IN_FLIGHT = 4
MAX_URLS = 50
MAX_HOPS = 2
BACKOFF_START = 10
MAX_ATTEMPTS = 3
TTL_PRESS = 7 * 86400
TTL_PRIMARY = 30 * 86400
TTL_ROBOTS = 86400
MAX_PRESS_FOLLOW = 3
MAX_PRIMARY_TRIES = 4
# A primary published long before the article is background, not the origin.
WINDOW_BEFORE_DAYS = 60
WINDOW_AFTER_DAYS = 3
# Outlets that link their source do so early. A candidate whose date cannot be checked
# is accepted as the primary only when it is linked within the first paragraphs.
EARLY_PARAGRAPHS = 4
# Access types a primary of each type can hold. A vendor report that discusses a leak
# site has not thereby obtained its information by actor statement.
ACCESS_BY_TYPE = {
    "vendor": {"telemetry", "ir_engagement", "sample_analysis", "osint"},
    "government": {"telemetry", "ir_engagement", "sample_analysis", "osint", "victim_statement"},
    "researcher": {"telemetry", "ir_engagement", "sample_analysis", "osint"},
    "victim": {"victim_statement"},
    "actor": {"actor_statement"},
}

SKILL_DIR = Path(__file__).resolve().parent.parent
SOURCE_TYPES = SKILL_DIR / "references" / "source-types.md"

PRIMARY_TYPES = {"vendor", "government", "victim", "actor", "researcher"}
INTERMEDIARY_TYPES = {"press", "aggregator", "social", "unknown"}

# --- pattern lists (documented in references/attribution-phrases.md) -----------------

ATTRIBUTION_PATTERNS = [
    r"\baccording to\b",
    r"\bresearchers (?:at|from|with)\b",
    r"\bin an? (?:new |recent |joint )?(?:report|blog post|advisory|analysis|write-up|technical write-up)\b",
    r"\b(?:said|wrote|noted|stated|explained|warned) in (?:a|an|its|their)\b",
    r"\bpublished by\b",
    r"\b(?:a|an|the) (?:joint )?advisory\b",
    r"\bdisclosed\b",
    r"\btold [A-Z]",
    r"\b(?:first )?(?:reported|spotted|discovered|uncovered|documented|detailed) by\b",
    r"\b(?:findings|data|telemetry) from\b",
    r"\bshared with\b",
    r"\b(?:8-K|SEC) filing\b|\bfiled with\b",
    r"\b(?:data )?leak site\b|\bposted on (?:its|their) [^.]{0,40}(?:site|blog|channel)\b",
]

UNATTRIBUTED_PATTERNS = [
    r"\bresearchers (?:say|said|have|warn)\b",
    r"\bsecurity experts\b|\bexperts (?:say|warn)\b",
    r"\bsources (?:say|said|familiar with)\b",
    r"\breports (?:suggest|indicate)\b|\bit is (?:believed|reported)\b",
]

CONFIDENCE_PATTERNS = [
    r"\bassess(?:es|ed|ment)?\b[^.]{0,40}\b(?:with )?(?:low|moderate|medium|high) confidence\b",
    r"\b(?:low|moderate|medium|high)[- ]confidence\b",
    r"\bwith (?:low|moderate|medium|high) confidence\b",
    r"\bconfidence (?:level|is)\b",
]

HEDGE_PATTERNS = [
    r"\blikely\b", r"\bunlikely\b", r"\bsuspected\b", r"\bpossibly\b", r"\bpotentially\b",
    r"\bprobably\b", r"\bmay have\b", r"\bmight\b", r"\bappears? to\b", r"\bwe believe\b",
    r"\bbelieved to\b", r"\bconsistent with\b", r"\boverlaps? with\b", r"\blinked to\b",
    r"\battributed\b", r"\bcannot confirm\b", r"\bcould not confirm\b", r"\bunconfirmed\b",
    r"\balleged(?:ly)?\b", r"\bclaim(?:s|ed)\b",
]

# Order matters: ties between access types resolve in this order.
ACCESS_PATTERNS = [
    ("telemetry", [
        r"\bour telemetry\b",
        r"\b(?:our|its|their) (?:own )?(?:sensors|telemetry|visibility|customer base|honeypots?)\b",
        r"\bwe (?:have )?observed\b", r"\bwe (?:have )?detected\b",
        r"\b(?:observed|detected|blocked) (?:by|in) (?:our|its)\b",
        r"\b(?:product|endpoint|network) telemetry\b",
    ]),
    ("ir_engagement", [
        r"\bincident response (?:engagement|investigation|case)s?\b",
        r"\bduring (?:an|a recent|our|the) (?:investigation|engagement|response)\b",
        r"\b(?:were|was) engaged (?:by|to)\b",
        r"\bresponded to (?:an|a|the) (?:incident|intrusion|breach)\b",
        r"\bforensic (?:analysis|investigation|examination)\b",
    ]),
    ("sample_analysis", [
        r"\bsamples? (?:was |were )?(?:submitted|uploaded|obtained|shared)\b",
        r"\buploaded to VirusTotal\b",
        r"\bwe (?:analy[sz]ed|reverse[- ]engineered)\b",
        r"\b(?:static|dynamic) analysis\b", r"\breverse engineering\b",
    ]),
    ("osint", [
        r"\bopen[- ]source (?:reporting|information|intelligence|research)\b",
        r"\bpublicly (?:available|reported)\b", r"\bpublic reporting\b",
        r"\bpreviously (?:reported|documented) by\b",
    ]),
    ("victim_statement", [
        r"\b(?:8-K|10-K|SEC) filing\b", r"\bnotification (?:letter|to affected)\b",
        r"\bwe (?:recently )?(?:discovered|identified|detected|became aware of) "
        r"(?:unauthorized|unauthorised|suspicious)\b",
        r"\bbreach notification\b",
    ]),
    ("actor_statement", [
        r"\bleak site\b", r"\bransom note\b",
        r"\bposted on (?:a|an|the) (?:forum|Telegram)\b", r"\bclaimed responsibility\b",
    ]),
]


def _compile(patterns):
    return [re.compile(p, re.IGNORECASE) for p in patterns]


RE_ATTRIBUTION = _compile(ATTRIBUTION_PATTERNS)
RE_UNATTRIBUTED = _compile(UNATTRIBUTED_PATTERNS)
RE_CONFIDENCE = _compile(CONFIDENCE_PATTERNS)
RE_HEDGE = _compile(HEDGE_PATTERNS)
RE_ACCESS = [(name, _compile(pats)) for name, pats in ACCESS_PATTERNS]
RE_SENTENCE = re.compile(r"(?<=[.!?])[\"”’)]*\s+(?=[\"“‘(]?[A-Z0-9])")
RE_ISO_DATE = re.compile(r"(\d{4})-(\d{2})-(\d{2})")
RE_URL_DATE = re.compile(r"/(20\d{2})/(\d{1,2})/(?:(\d{1,2})/)?")
RE_PROMO_QUERY = re.compile(r"(?:^|&)(?:utm_[a-z]+|ref|campaign|gclid|fbclid)=", re.IGNORECASE)
RE_PROMO_PATH = re.compile(
    r"^/(?:[a-z]{2}(?:-[a-z]{2})?/)?(?:enterprise|pricing|plans|demo|request-demo|get-demo|"
    r"contact|contact-us|contact-sales|signup|sign-up|register|login|trial|free-trial|"
    r"products?|platform|solutions|services|why-[\w-]+|about|about-us|careers|partners|"
    r"webinars?|events|subscribe|newsletter|download)(?:/|$)", re.IGNORECASE)


class Refused(Exception):
    pass


# --- source-types table --------------------------------------------------------------

class SourceTable:
    """Lookup table parsed from references/source-types.md. Longest prefix wins."""

    COLUMNS = ["domain", "organisation", "type", "access_profile", "interest", "notes"]

    def __init__(self, path=SOURCE_TYPES):
        self.rows = []
        self._parse(Path(path).read_text(encoding="utf-8"))
        self.by_host = {}
        for row in self.rows:
            self.by_host.setdefault(row["host"], []).append(row)
        for rows in self.by_host.values():
            rows.sort(key=lambda r: len(r["path"]), reverse=True)

    def _parse(self, text):
        in_table = False
        for line in text.splitlines():
            if not line.startswith("|"):
                in_table = False
                continue
            cells = [c.strip() for c in line.strip().strip("|").split("|")]
            if [c.lower() for c in cells] == self.COLUMNS:
                in_table = True
                continue
            if not in_table or set(cells[0]) <= {"-", ":"} or len(cells) != len(self.COLUMNS):
                continue
            row = dict(zip(self.COLUMNS, cells))
            host, _, path = row["domain"].lower().partition("/")
            row["host"] = host
            row["path"] = "/" + path if path else ""
            row["names"] = [n.strip() for n in row["organisation"].split(" / ") if n.strip()]
            row["is_primary_capable"] = not row["notes"].lower().startswith("not a primary")
            self.rows.append(row)

    def lookup(self, url):
        parts = urllib.parse.urlsplit(url)
        host = (parts.hostname or "").lower()
        path = parts.path or "/"
        if host.endswith(".onion"):
            return {"domain": host, "organisation": None, "type": "actor",
                    "access_profile": "actor_statement", "interest": "actor",
                    "notes": "onion host: built-in rule", "is_primary_capable": True,
                    "matched": "rule:onion"}
        labels = host.split(".")
        for i in range(len(labels) - 1):
            candidate = ".".join(labels[i:])
            for row in self.by_host.get(candidate, []):
                if not row["path"] or path.lower().startswith(row["path"]):
                    out = {k: row[k] for k in self.COLUMNS}
                    out["organisation"] = row["names"][0]
                    out["is_primary_capable"] = row["is_primary_capable"]
                    out["matched"] = row["domain"]
                    return out
        return {"domain": host, "organisation": None, "type": "unknown",
                "access_profile": "undisclosed", "interest": None,
                "notes": "host not in source-types.md", "is_primary_capable": False,
                "matched": None}

    def names(self):
        """(compiled name pattern, row) for every organisation name, longest first."""
        out = []
        for row in self.rows:
            for name in row["names"]:
                if len(name) < 3:
                    continue
                out.append((re.compile(r"(?<![\w-])" + re.escape(name) + r"(?![\w-])"), name, row))
        out.sort(key=lambda t: len(t[1]), reverse=True)
        return out


# --- fetcher ---------------------------------------------------------------------------

class _HTTPSConnection(http.client.HTTPSConnection):
    def connect(self):
        super().connect()
        self.sock.settimeout(READ_TIMEOUT)


class _HTTPConnection(http.client.HTTPConnection):
    def connect(self):
        super().connect()
        self.sock.settimeout(READ_TIMEOUT)


class _HTTPSHandler(urllib.request.HTTPSHandler):
    def https_open(self, req):
        return self.do_open(_HTTPSConnection, req)


class _HTTPHandler(urllib.request.HTTPHandler):
    def http_open(self, req):
        return self.do_open(_HTTPConnection, req)


class FetchResult:
    def __init__(self, url, status, body=None, content_type=None, final_url=None,
                 error=None, from_cache=False):
        self.url = url
        self.status = status          # ok | pdf | blocked_by_robots | http_error | ...
        self.body = body
        self.content_type = content_type
        self.final_url = final_url or url
        self.error = error
        self.from_cache = from_cache

    @property
    def ok(self):
        return self.status == "ok"

    def log(self):
        return {"url": self.url, "final_url": self.final_url, "status": self.status,
                "from_cache": self.from_cache, "reason": self.error}


class Fetcher:
    def __init__(self, table, cache_dir=None, use_cache=True, fixtures=None):
        self.table = table
        self.use_cache = use_cache
        self.cache_dir = Path(cache_dir or os.environ.get("CTI_SKILLS_CACHE")
                              or Path.home() / ".cache" / "cti-skills" / "provenance")
        self.fixtures = fixtures or {}
        self.opener = urllib.request.build_opener(_HTTPHandler, _HTTPSHandler)
        self.host_locks = {}
        self.host_last = {}
        self.skipped_hosts = {}
        self.robots = {}
        self.lock = threading.Lock()
        self.global_slots = threading.Semaphore(MAX_HOSTS_IN_FLIGHT)
        self.fetch_count = 0
        self.log = []

    # cache ---------------------------------------------------------------------------
    def _cache_paths(self, url):
        key = hashlib.sha256(url.encode("utf-8")).hexdigest()
        return self.cache_dir / (key + ".body"), self.cache_dir / (key + ".json")

    def _cache_read(self, url):
        body_p, meta_p = self._cache_paths(url)
        if not (self.use_cache and body_p.exists() and meta_p.exists()):
            return None, None
        try:
            return body_p.read_bytes(), json.loads(meta_p.read_text())
        except (OSError, ValueError):
            return None, None

    def _cache_write(self, url, body, meta):
        if not self.use_cache:
            return
        try:
            self.cache_dir.mkdir(parents=True, exist_ok=True)
            body_p, meta_p = self._cache_paths(url)
            body_p.write_bytes(body)
            meta_p.write_text(json.dumps(meta))
        except OSError:
            pass

    def text_path(self, url):
        key = hashlib.sha256(url.encode("utf-8")).hexdigest()
        return self.cache_dir / (key + ".txt")

    def _ttl(self, url):
        if url.endswith("/robots.txt"):
            return TTL_ROBOTS
        return TTL_PRIMARY if self.table.lookup(url)["type"] in PRIMARY_TYPES else TTL_PRESS

    # robots --------------------------------------------------------------------------
    def _allowed(self, url):
        parts = urllib.parse.urlsplit(url)
        origin = f"{parts.scheme}://{parts.netloc}"
        with self.lock:
            rp = self.robots.get(origin)
        if rp is None:
            res = self._get(origin + "/robots.txt", check_robots=False)
            rp = urllib.robotparser.RobotFileParser()
            if res.ok:
                rp.parse(res.body.decode("utf-8", "replace").splitlines())
            elif res.status == "http_error" and res.error and res.error.startswith("HTTP 4"):
                rp.parse([])                                  # RFC 9309: 4xx means no restrictions
            else:
                rp.parse(["User-agent: *", "Disallow: /"])    # unreachable: assume disallow
            with self.lock:
                self.robots[origin] = rp
        return rp.can_fetch(ROBOTS_AGENT, url)

    # fetch ---------------------------------------------------------------------------
    def fetch(self, url):
        res = self._get(url, check_robots=True)
        with self.lock:
            self.log.append(res.log())
        return res

    def _get(self, url, check_robots):
        if url in self.fixtures:
            return self._from_fixture(url)
        if self.fixtures:
            return FetchResult(url, "http_error", error="HTTP 404 (not in fixture map)")
        parts = urllib.parse.urlsplit(url)
        if parts.scheme not in ("http", "https") or not parts.hostname:
            return FetchResult(url, "unsupported_url", error="only http and https are fetched")
        host = parts.hostname.lower()
        if host.endswith(".onion"):
            return FetchResult(url, "not_fetched", error="onion hosts are never fetched")
        if host in self.skipped_hosts:
            return FetchResult(url, "host_skipped", error=self.skipped_hosts[host])

        body, meta = self._cache_read(url)
        now = time.time()
        if meta and now - meta.get("fetched_at", 0) < self._ttl(url):
            return self._result_from_cache(url, body, meta)

        if check_robots and not self._allowed(url):
            return FetchResult(url, "blocked_by_robots",
                               error="robots.txt disallows this path; paste the text to continue")

        with self.lock:
            if check_robots:
                self.fetch_count += 1
                if self.fetch_count > MAX_URLS:
                    return FetchResult(url, "cap_reached",
                                       error=f"hard cap of {MAX_URLS} URLs per invocation reached")
            host_lock = self.host_locks.setdefault(host, threading.Lock())

        with self.global_slots, host_lock:
            return self._request(url, host, body, meta)

    def _request(self, url, host, cached_body, cached_meta):
        delay = BACKOFF_START
        for attempt in range(1, MAX_ATTEMPTS + 1):
            wait = HOST_SPACING - (time.time() - self.host_last.get(host, 0))
            if wait > 0:
                time.sleep(wait)
            req = urllib.request.Request(url, headers={
                "User-Agent": USER_AGENT,
                "Accept": "text/html,application/xhtml+xml,application/pdf;q=0.8,*/*;q=0.5",
                "Accept-Language": "en",
            })
            if cached_meta:
                if cached_meta.get("etag"):
                    req.add_header("If-None-Match", cached_meta["etag"])
                if cached_meta.get("last_modified"):
                    req.add_header("If-Modified-Since", cached_meta["last_modified"])
            try:
                with self.opener.open(req, timeout=CONNECT_TIMEOUT) as resp:
                    self.host_last[host] = time.time()
                    ctype = (resp.headers.get("Content-Type") or "").split(";")[0].strip().lower()
                    final_url = resp.geturl()
                    if ctype == "application/pdf" or final_url.lower().split("?")[0].endswith(".pdf"):
                        return FetchResult(url, "pdf", content_type=ctype, final_url=final_url,
                                           error="PDF: route to the pdf skill")
                    body = resp.read(MAX_BYTES + 1)
                    if len(body) > MAX_BYTES:
                        return FetchResult(url, "size_limit", final_url=final_url,
                                           error="response larger than 5 MB")
                    meta = {"fetched_at": time.time(), "etag": resp.headers.get("ETag"),
                            "last_modified": resp.headers.get("Last-Modified"),
                            "content_type": ctype, "final_url": final_url}
                    self._cache_write(url, body, meta)
                    return FetchResult(url, "ok", body, ctype, final_url)
            except urllib.error.HTTPError as e:
                self.host_last[host] = time.time()
                if e.code == 304 and cached_meta:
                    cached_meta["fetched_at"] = time.time()
                    self._cache_write(url, cached_body, cached_meta)
                    return self._result_from_cache(url, cached_body, cached_meta)
                if e.code in (429, 503):
                    if attempt == MAX_ATTEMPTS:
                        reason = f"HTTP {e.code} after {MAX_ATTEMPTS} attempts; host skipped for this run"
                        self.skipped_hosts[host] = reason
                        return FetchResult(url, "rate_limited", error=reason)
                    retry_after = e.headers.get("Retry-After") if e.headers else None
                    try:
                        pause = min(float(retry_after), 120.0) if retry_after else delay
                    except ValueError:
                        pause = delay
                    time.sleep(pause)
                    delay *= 2
                    continue
                return FetchResult(url, "http_error", error=f"HTTP {e.code}")
            except (urllib.error.URLError, OSError, http.client.HTTPException) as e:
                self.host_last[host] = time.time()
                return FetchResult(url, "network_error", error=str(getattr(e, "reason", e)))
        return FetchResult(url, "network_error", error="exhausted attempts")

    def _result_from_cache(self, url, body, meta):
        ctype = meta.get("content_type")
        if ctype == "application/pdf":
            return FetchResult(url, "pdf", content_type=ctype, final_url=meta.get("final_url"),
                               error="PDF: route to the pdf skill", from_cache=True)
        return FetchResult(url, "ok", body, ctype, meta.get("final_url"), from_cache=True)

    def _from_fixture(self, url):
        entry = self.fixtures[url]
        if isinstance(entry, dict):
            return FetchResult(url, entry.get("status", "http_error"), error=entry.get("reason"))
        try:
            return FetchResult(url, "ok", Path(entry).read_bytes(), "text/html", url, from_cache=True)
        except OSError as e:
            return FetchResult(url, "network_error", error=f"fixture unreadable: {e}")


# --- HTML extraction -------------------------------------------------------------------

VOID_TAGS = {"area", "base", "br", "col", "embed", "hr", "img", "input", "link", "meta",
             "param", "source", "track", "wbr"}
# Removed wherever they appear.
SKIP_TAGS = {"nav", "footer", "aside", "script", "style", "form", "noscript", "svg",
             "iframe", "figure", "button", "select", "template"}
# Class, id or role tokens that suggest boilerplate. A matching subtree inside the
# article is dropped only when it is link-dense or a minor share of the article text,
# so a wrapper such as "sidebar-page-main" around the whole story is kept.
BOILERPLATE = re.compile(
    r"(?:^|[\s_-])(?:related|share|sharing|social|sidebar|footer|nav|navbar|menu|comments?|"
    r"newsletter|promo|widget|breadcrumbs?|popular|trending|recommended|subscribe|"
    r"advert|ads?|cookie|author-bio|byline|more-stories|read-more|also-read|latest|"
    r"webinars?|sponsored)(?:$|[\s_-])",
    re.IGNORECASE)
BLOCK_TAGS = {"p", "li", "blockquote", "h1", "h2", "h3", "h4", "td"}
LINK_DENSE = 0.4
MINOR_SHARE = 0.25


class Node:
    __slots__ = ("tag", "attrs", "children", "parent", "_text_len", "_link_len")

    def __init__(self, tag, attrs=None, parent=None):
        self.tag = tag
        self.attrs = attrs or {}
        self.children = []
        self.parent = parent
        self._text_len = None
        self._link_len = None

    def marker(self):
        return " ".join(filter(None, [self.attrs.get("class"), self.attrs.get("id"),
                                      self.attrs.get("role")]))

    def iter(self):
        yield self
        for child in self.children:
            if isinstance(child, Node):
                yield from child.iter()

    def measure(self):
        """(text length, text length inside links) for the subtree."""
        if self._text_len is None:
            text = link = 0
            for child in self.children:
                if isinstance(child, Node):
                    t, l = child.measure()
                    text += t
                    link += t if child.tag == "a" else l
                else:
                    text += len(child.strip())
            self._text_len, self._link_len = text, link
        return self._text_len, self._link_len


class ArticleParser(HTMLParser):
    """Builds a small tree, then selects the article body by paragraph-text score."""

    def __init__(self, base_url):
        super().__init__(convert_charrefs=True)
        self.base_url = base_url
        self.root = Node("root")
        self.current = self.root
        self.meta = {}
        self.jsonld = []
        self.title = None
        self.time_datetimes = []
        self._capture = None        # "title" | "jsonld" | "skip"
        self._capture_tag = None
        self._buffer = []

    def handle_starttag(self, tag, attrs):
        a = dict(attrs)
        if self._capture:
            return
        if tag == "meta":
            key = (a.get("property") or a.get("name") or a.get("itemprop") or "").lower()
            if key and a.get("content"):
                self.meta.setdefault(key, a["content"].strip())
            return
        if tag == "time" and a.get("datetime"):
            self.time_datetimes.append(a["datetime"])
        if tag in VOID_TAGS:
            return
        if tag == "title" and self.title is None:
            self._capture, self._capture_tag, self._buffer = "title", tag, []
            return
        if tag == "script":
            kind = "jsonld" if (a.get("type") or "").lower() == "application/ld+json" else "skip"
            self._capture, self._capture_tag, self._buffer = kind, tag, []
            return
        if tag == "style":
            self._capture, self._capture_tag, self._buffer = "skip", tag, []
            return
        if tag == "p" and self.current.tag == "p":
            self.current = self.current.parent          # implicit </p>
        node = Node(tag, a, self.current)
        self.current.children.append(node)
        self.current = node

    def handle_endtag(self, tag):
        if self._capture:
            if tag == self._capture_tag:
                if self._capture == "title":
                    self.title = " ".join("".join(self._buffer).split())
                elif self._capture == "jsonld":
                    self.jsonld.append("".join(self._buffer))
                self._capture = None
            return
        node = self.current
        while node is not None and node.tag != tag:
            node = node.parent
        if node is not None and node.parent is not None:
            self.current = node.parent

    def handle_data(self, data):
        if self._capture:
            self._buffer.append(data)
        else:
            self.current.children.append(data)

    # selection -----------------------------------------------------------------------
    def _prune(self, node, in_article=False):
        kept = []
        for child in node.children:
            if isinstance(child, Node):
                if child.tag in SKIP_TAGS and not (
                        child.tag == "figure" and any(n.tag == "table" for n in child.iter())):
                    continue                        # a figure that wraps a table is kept
                if child.tag == "header" and not in_article:
                    continue
                self._prune(child, in_article or child.tag == "article")
            kept.append(child)
        node.children = kept

    def _content_root(self):
        scores = {}
        held = {}                   # <article>/<main> -> paragraph text inside it
        page_total = 0
        for node in self.root.iter():
            if node.tag != "p" or node.parent is None:
                continue
            length = node.measure()[0]
            if length < 60:
                continue
            page_total += length
            parent = node.parent
            scores[parent] = scores.get(parent, 0) + length
            if parent.parent is not None:
                scores[parent.parent] = scores.get(parent.parent, 0) + length / 2
            up = parent
            while up is not None:
                if up.tag in ("article", "main"):
                    held[up] = held.get(up, 0) + length
                up = up.parent
        if not scores:
            return self.root
        best = max(scores, key=scores.get)
        # Widen to an enclosing <article> or <main> when it holds most of the same text,
        # or when the body is split across sibling sections and the enclosing element
        # holds most of the paragraph text on the page.
        node = best.parent
        while node is not None and node.parent is not None:
            if node.tag in ("article", "main") and (
                    best.measure()[0] >= 0.6 * node.measure()[0]
                    or held.get(node, 0) >= 0.6 * page_total):
                return node
            node = node.parent
        return best

    def blocks(self):
        self._prune(self.root)
        root = self._content_root()
        total = max(root.measure()[0], 1)
        out = []
        self._collect(root, root, total, out)
        if not any(b["tag"] == "h1" for b in out):
            for node in self.root.iter():
                if node.tag == "h1":
                    text, _ = self._flatten(node)
                    if text:
                        out.insert(0, {"tag": "h1", "text": text, "links": []})
                    break
        return out

    def _collect(self, node, root, total, out):
        for child in node.children:
            if not isinstance(child, Node):
                continue
            marker = child.marker()
            if child is not root and marker and BOILERPLATE.search(marker):
                text_len, link_len = child.measure()
                if text_len == 0 or link_len / text_len >= LINK_DENSE or text_len / total < MINOR_SHARE:
                    continue
            if child.tag == "tr":
                cells, links = [], []
                for cell in child.children:
                    if isinstance(cell, Node) and cell.tag in ("td", "th"):
                        text, found = self._flatten(cell)
                        shift = len(" | ".join(cells + [""])) if cells else 0
                        for link in found:
                            link["offset"] += shift
                        cells.append(text)
                        links.extend(found)
                if any(cells):
                    out.append({"tag": "tr", "text": " | ".join(cells), "links": links})
                continue
            if child.tag in BLOCK_TAGS and not any(
                    n.tag in BLOCK_TAGS for n in child.iter() if n is not child):
                text, links = self._flatten(child)
                if text:
                    out.append({"tag": child.tag, "text": text, "links": links})
            else:
                self._collect(child, root, total, out)

    def _flatten(self, node):
        parts, links = [], []

        def walk(n, link):
            for child in n.children:
                if isinstance(child, Node):
                    if child.tag == "a" and child.attrs.get("href") and link is None:
                        entry = {"href": urllib.parse.urljoin(self.base_url,
                                                              child.attrs["href"].strip()),
                                 "start": len("".join(parts)), "text": []}
                        walk(child, entry)
                        entry["text"] = " ".join("".join(entry["text"]).split())
                        links.append(entry)
                    else:
                        if child.tag == "br":
                            parts.append(" ")
                        walk(child, link)
                else:
                    parts.append(child)
                    if link is not None:
                        link["text"].append(child)

        walk(node, None)
        raw = "".join(parts)
        for link in links:
            link["offset"] = len(" ".join(raw[:link["start"]].split()))
        return " ".join(raw.split()), links


def iso_date(value):
    if not value:
        return None
    m = RE_ISO_DATE.search(str(value))
    if not m:
        return None
    try:
        return dt.date(int(m.group(1)), int(m.group(2)), int(m.group(3))).isoformat()
    except ValueError:
        return None


def _walk_jsonld(node):
    if isinstance(node, dict):
        yield node
        for v in node.values():
            yield from _walk_jsonld(v)
    elif isinstance(node, list):
        for v in node:
            yield from _walk_jsonld(v)


def split_sentences(text):
    return [s.strip() for s in RE_SENTENCE.split(text) if s.strip()]


def sentence_at(text, offset):
    pos = 0
    for sentence in split_sentences(text):
        start = text.find(sentence, pos)
        if start < 0:
            continue
        end = start + len(sentence)
        if start <= offset <= end:
            return sentence
        pos = end
    return text if len(text) <= 600 else text[:600]


class Page:
    """Extracted fields for one fetched document."""

    def __init__(self, url, body_bytes):
        self.url = url
        charset = "utf-8"
        head = body_bytes[:4096].decode("ascii", "replace")
        m = re.search(r"charset=[\"']?([\w-]+)", head, re.IGNORECASE)
        if m:
            charset = m.group(1)
        try:
            markup = body_bytes.decode(charset, "replace")
        except LookupError:
            markup = body_bytes.decode("utf-8", "replace")
        parser = ArticleParser(url)
        try:
            parser.feed(markup)
            parser.close()
        except Exception:                # malformed markup: keep what was parsed so far
            pass
        try:
            self.blocks = parser.blocks()
        except RecursionError:
            self.blocks = []
        self.meta = parser.meta
        self.jsonld = []
        for raw in parser.jsonld:
            try:
                self.jsonld.extend(_walk_jsonld(json.loads(raw)))
            except ValueError:
                continue
        self._title_tag = parser.title
        self._times = parser.time_datetimes

    @property
    def title(self):
        for node in self.jsonld:
            if node.get("headline"):
                return html.unescape(str(node["headline"])).strip()
        for key in ("og:title", "twitter:title"):
            if self.meta.get(key):
                return self.meta[key]
        for b in self.blocks:
            if b["tag"] == "h1":
                return b["text"]
        return self._title_tag or None

    @property
    def date_published(self):
        for node in self.jsonld:
            d = iso_date(node.get("datePublished"))
            if d:
                return d
        for key in ("article:published_time", "og:article:published_time", "date", "pubdate",
                    "publish-date", "publish_date", "parsely-pub-date", "dc.date.issued",
                    "dc.date", "dcterms.created", "sailthru.date", "datepublished"):
            d = iso_date(self.meta.get(key))
            if d:
                return d
        for value in self._times:
            d = iso_date(value)
            if d:
                return d
        return None

    @property
    def paragraphs(self):
        return [b for b in self.blocks if b["tag"] in ("p", "li", "blockquote", "td", "tr")]

    def text(self):
        return "\n\n".join(b["text"] for b in self.blocks)

    def sentences(self):
        for index, block in enumerate(self.paragraphs):
            for sentence in split_sentences(block["text"]):
                yield index, sentence

    def body_links(self):
        for index, block in enumerate(self.paragraphs):
            for link in block["links"]:
                parts = urllib.parse.urlsplit(link["href"])
                if parts.scheme not in ("http", "https"):
                    continue
                href = urllib.parse.urlunsplit(parts._replace(fragment=""))
                yield {"href": href, "text": link["text"], "paragraph": index,
                       "sentence": sentence_at(block["text"], link["offset"])}


def url_date(url):
    m = RE_URL_DATE.search(urllib.parse.urlsplit(url).path)
    if not m:
        return None
    try:
        return dt.date(int(m.group(1)), int(m.group(2)), int(m.group(3) or 1)).isoformat()
    except ValueError:
        return None


def within_window(article_date, primary_date, month_only=False):
    a = dt.date.fromisoformat(article_date)
    p = dt.date.fromisoformat(primary_date)
    before = WINDOW_BEFORE_DAYS + (31 if month_only else 0)
    return -WINDOW_AFTER_DAYS - (31 if month_only else 0) <= (a - p).days <= before


RE_NEGATION = re.compile(r"\b(?:no|not|never|without|neither|nor)\b|n['’]t\b", re.IGNORECASE)


def matches_affirmed(patterns, text):
    """Like matches_any, but a match that follows a negation in the same clause is ignored:
    "we did not observe a ransom note" is not evidence of access to an actor's statement."""
    for pattern in patterns:
        for m in pattern.finditer(text):
            clause = re.split(r"[,;:]", text[:m.start()])[-1]
            if not RE_NEGATION.search(clause[-60:]):
                return True
    return False


def matches_any(patterns, text):
    return any(p.search(text) for p in patterns)


def same_org(a, b):
    return bool(a and b and a == b)


# --- resolution ------------------------------------------------------------------------

class Resolver:
    def __init__(self, table, fetcher, max_hops=MAX_HOPS, today=None):
        self.table = table
        self.fetcher = fetcher
        self.max_hops = max_hops
        self.today = today or dt.date.today()
        self.names = table.names()

    def resolve(self, url):
        out = {
            "input_url": url, "status": "unresolved", "provenance_basis": "script-resolved",
            "chain": [], "primary_source": None, "other_primaries": [], "primary_cites": [],
            "background_references": [], "unclassified_candidates": [],
            "circular_citations": False,
            "recency": {"date_observed": None, "date_published": None, "age_days": None,
                        "superseded_by": None},
            "attribution_sentences": [], "unattributed_sourcing": [],
            "unresolved": [{"field": "recency.date_observed",
                            "reason": "when the activity happened is stated in prose, if at "
                                      "all; recorded by /claim-extraction"}],
            "summary": None,
        }
        self._walk(url, 0, {url}, out, via=None)
        primary = out["primary_source"]
        if primary and primary.get("url"):
            out["status"] = "resolved"
            published = primary.get("date_published")
            out["recency"]["date_published"] = published
            if published:
                out["recency"]["age_days"] = (self.today - dt.date.fromisoformat(published)).days
        elif not any(u["field"] == "primary_source" for u in out["unresolved"]):
            out["unresolved"].append({"field": "primary_source",
                                      "reason": "no originating document located"})
        out["unclassified_candidates"].sort(
            key=lambda c: (not c["in_attribution_sentence"], c["paragraph"]))
        del out["unclassified_candidates"][10:]
        out["summary"] = self._summary(out)
        return out

    def _hop(self, url, cls, page=None, fidelity=None, notes=None, anchor=None):
        return {"org": cls["organisation"], "type": cls["type"], "url": url,
                "date_published": page.date_published if page else None,
                "title": page.title if page else None,
                "fidelity": fidelity, "anchor_in_previous_hop": anchor, "notes": notes}

    def _read_primary(self, url, cls, anchor):
        """Fetch a primary-type document. Returns (primary, hop)."""
        res = self.fetcher.fetch(url)
        page = Page(res.final_url, res.body) if res.ok else None
        note = None if res.ok else f"primary located but not read: {res.error}"
        hop = self._hop(url, cls, page, anchor=anchor, notes=note,
                        fidelity=None if res.ok else "unresolved")
        return self._primary(url, cls, page, res), hop, page

    def _cites(self, page, url, cls, out):
        """Other originating organisations the primary itself links to or names as a source."""
        if page is None:
            return
        self._attribution(page, url, out)
        candidates, _, _ = self._candidates(page, url, cls, set(), out)
        # From a primary, keep an unclassified host only where it is named as a source.
        out["unclassified_candidates"] = [
            c for c in out["unclassified_candidates"]
            if c["found_in"] != url or c["in_attribution_sentence"]]
        seen = {c["org"] for c in out["primary_cites"]}
        for cand in candidates:
            org = cand["cls"]["organisation"]
            if org in seen or len(out["primary_cites"]) >= 10:
                continue
            seen.add(org)
            out["primary_cites"].append({
                "org": org, "type": cand["cls"]["type"], "url": cand["href"],
                "anchor_in_primary": cand["sentence"],
                "in_attribution_sentence": bool(cand["attributed"]), "read": False,
                "note": "linked from the primary; not read. A claim the primary relays from "
                        "this organisation belongs to it and needs its own chain"})
        relayed = [a for a in out["attribution_sentences"] if a["document_url"] == url]
        if relayed:
            out["primary_relays"] = {
                "sentences": len(relayed),
                "note": "the primary attributes at least one statement to another source. "
                        "Read attribution_sentences for this document: the claims in them "
                        "are relayed, and this organisation's access does not apply to them"}

    def _walk(self, url, depth, visited, out, via):
        """Returns True once a primary has been recorded."""
        cls = self.table.lookup(url)
        if cls["type"] in PRIMARY_TYPES and cls["is_primary_capable"]:
            primary, hop, page = self._read_primary(url, cls, via)
            out["chain"].append(hop)
            out["primary_source"] = primary
            self._cites(page, url, cls, out)
            if primary["resolution"] == "linked_not_read":
                out["unresolved"].append({"field": "primary_source.document", "url": url,
                                          "reason": hop["notes"]})
            return True

        res = self.fetcher.fetch(url)
        if not res.ok:
            out["chain"].append(self._hop(url, cls, fidelity="unresolved", anchor=via,
                                          notes=f"fetch failed: {res.error}"))
            out["unresolved"].append({"field": "chain", "url": url, "reason": res.error})
            return False

        page = Page(res.final_url, res.body)
        out["chain"].append(self._hop(url, cls, page, anchor=via))
        hop_index = len(out["chain"]) - 1
        self._attribution(page, url, out)

        candidates, press_links, circular = self._candidates(page, url, cls, visited, out)
        if self._pick_primary(candidates, page, out):
            return True

        if depth + 1 < self.max_hops:
            for link in press_links[:MAX_PRESS_FOLLOW]:
                visited.add(link["href"])
                mark = len(out["unresolved"])
                if self._walk(link["href"], depth + 1, visited, out, via=link["sentence"]):
                    return True
                del out["chain"][hop_index + 1:]      # dead end: drop the failed branch
                del out["unresolved"][mark:]

        out["chain"][hop_index]["fidelity"] = "unresolved"
        if out["primary_source"] is not None:
            return False
        named = [a for a in out["attribution_sentences"]
                 if a["document_url"] == url and a["organisations"] and not a["links"]]
        if named:
            org = named[0]["organisations"][0]
            out["primary_source"] = {
                "org": org["org"], "type": org["type"], "title": None, "date_published": None,
                "url": None, "access": "undisclosed", "access_evidence": [],
                "stated_confidence": "not stated", "confidence_evidence": [], "hedges": [],
                "interest": org["interest"], "resolution": "named_not_linked",
                "named_in": named[0]["sentence"],
            }
            reason = (f"{org['org']} is named as the source but no document is linked; "
                      "the primary was not read")
        elif circular or out["circular_citations"]:
            reason = "circular press citations, no primary found"
        elif press_links and depth + 1 >= self.max_hops:
            reason = f"only press links found and the hop limit ({self.max_hops}) was reached"
        else:
            reason = "no link to a vendor, government, victim, actor or researcher document"
        extras = []
        if any(b["found_in"] == url for b in out["background_references"]):
            extras.append("linked primary-type documents were treated as background")
        cited = sorted({urllib.parse.urlsplit(c["url"]).hostname
                        for c in out["unclassified_candidates"] if c["found_in"] == url})
        if cited:
            extras.append("the article cites " + ", ".join(cited[:4])
                          + ", which the script does not classify")
        out["unresolved"].append({"field": "primary_source", "url": url,
                                  "reason": "; ".join([reason] + extras)})
        return False

    def _pick_primary(self, candidates, page, out):
        """Read the top candidates. Prefer one whose date places it at the article's origin."""
        article_date = page.date_published
        verified, unverified, seen_orgs = [], [], set()
        tries = 0
        for cand in candidates:
            org = cand["cls"]["organisation"]
            if org in seen_orgs:
                continue
            hint = url_date(cand["href"])
            if article_date and hint and not within_window(article_date, hint, month_only=True):
                self._background(cand, hint, page, out,
                                 f"published more than {WINDOW_BEFORE_DAYS} days before the "
                                 "article (date in the URL)")
                continue
            if tries >= MAX_PRIMARY_TRIES:
                seen_orgs.add(org)
                out["other_primaries"].append({
                    "org": org, "type": cand["cls"]["type"], "url": cand["href"],
                    "anchor_in_article": cand["sentence"], "read": False,
                    "note": "linked from the article; not read (per-article read limit)"})
                continue
            tries += 1
            primary, hop, ppage = self._read_primary(cand["href"], cand["cls"], cand["sentence"])
            published = primary["date_published"]
            if article_date and published and not within_window(article_date, published):
                self._background(cand, published, page, out,
                                 f"published more than {WINDOW_BEFORE_DAYS} days before the "
                                 "article (page metadata)")
            elif published and article_date:
                seen_orgs.add(org)
                verified.append((cand, primary, hop, ppage))
            elif cand["early"]:
                seen_orgs.add(org)
                primary["date_check"] = ("not verified: the script could not compare "
                                         "publication dates, so it could not confirm this "
                                         "document is the origin of the article")
                unverified.append((cand, primary, hop, ppage))
            else:
                self._background(cand, None, page, out,
                                 "publication date could not be checked and the link sits "
                                 f"below paragraph {EARLY_PARAGRAPHS} of the article")
        accepted = verified + unverified
        if not accepted:
            return False
        cand, primary, hop, ppage = accepted[0]
        out["chain"].append(hop)
        out["primary_source"] = primary
        self._cites(ppage, cand["href"], cand["cls"], out)
        if primary["resolution"] == "linked_not_read":
            out["unresolved"].append({"field": "primary_source.document", "url": cand["href"],
                                      "reason": hop["notes"]})
        read = []
        for cand, other, _, _ in accepted[1:]:
            other["anchor_in_article"] = cand["sentence"]
            other["read"] = other["resolution"] == "linked"
            other["note"] = ("linked from the same article. Claims found in this document "
                             "belong to this organisation. Independence from the first "
                             "primary is not established by the link alone")
            read.append(other)
        out["other_primaries"] = read + out["other_primaries"]
        return True

    def _background(self, cand, date, page, out, why):
        out["background_references"].append({
            "org": cand["cls"]["organisation"], "url": cand["href"], "date": date,
            "found_in": page.url, "anchor_in_article": cand["sentence"],
            "note": f"{why}; treated as background, not as the originating source"})

    def _candidates(self, page, url, cls, visited, out):
        candidates, press, circular = [], [], False
        seen = set()
        for link in page.body_links():
            href = link["href"]
            if href in seen or href == url:
                continue
            seen.add(href)
            lcls = self.table.lookup(href)
            if same_org(lcls["organisation"], cls["organisation"]):
                continue
            parts = urllib.parse.urlsplit(href)
            path = parts.path.strip("/")
            if not path or RE_PROMO_QUERY.search(parts.query) or RE_PROMO_PATH.match(parts.path):
                continue                          # homepage, tracked or marketing link
            attributed = matches_any(RE_ATTRIBUTION, link["sentence"])
            early = link["paragraph"] < EARLY_PARAGRAPHS
            if lcls["type"] in PRIMARY_TYPES and lcls["is_primary_capable"]:
                link.update(cls=lcls, early=early, attributed=attributed,
                            rank=(0 if early and attributed else 1 if early else
                                  2 if attributed else 3, link["paragraph"]))
                candidates.append(link)
            elif lcls["type"] == "press":
                if href in visited:
                    circular = out["circular_citations"] = True
                    continue
                press.append(link)
            elif lcls["type"] in ("unknown", "social"):
                out["unclassified_candidates"].append({
                    "url": href, "type": lcls["type"], "found_in": url,
                    "paragraph": link["paragraph"] + 1,
                    "anchor_in_article": link["sentence"],
                    "in_attribution_sentence": attributed,
                    "note": ("social post: the account holder is the source; identity not "
                             "verified by this script" if lcls["type"] == "social" else
                             "host not in source-types.md; not classified and not followed. "
                             "Add a row if this organisation is a known source")})
        candidates.sort(key=lambda c: c["rank"])
        press.sort(key=lambda l: (0 if matches_any(RE_ATTRIBUTION, l["sentence"]) else 1,
                                  l["paragraph"]))
        return candidates, press, circular

    def _named_orgs(self, sentence, own):
        """Organisations from the table named in a sentence. Longest name claims its span."""
        claimed, orgs, seen = [], [], set()
        for pattern, name, row in self.names:
            for m in pattern.finditer(sentence):
                if any(m.start() < e and s < m.end() for s, e in claimed):
                    continue
                claimed.append((m.start(), m.end()))
                org = row["names"][0]
                if org == own or org in seen:
                    continue
                seen.add(org)
                orgs.append({"org": org, "matched_name": name, "type": row["type"],
                             "interest": row["interest"], "position": m.start()})
        orgs.sort(key=lambda o: o.pop("position"))
        return orgs

    def _attribution(self, page, url, out):
        links_by_sentence = {}
        for link in page.body_links():
            links_by_sentence.setdefault(link["sentence"], []).append(link["href"])
        own = self.table.lookup(url)["organisation"]
        for index, sentence in page.sentences():
            if matches_any(RE_ATTRIBUTION, sentence):
                orgs = self._named_orgs(sentence, own)
                if orgs or sentence in links_by_sentence:
                    out["attribution_sentences"].append({
                        "document_url": url, "location": f"paragraph {index + 1}",
                        "sentence": sentence, "organisations": orgs,
                        "links": links_by_sentence.get(sentence, [])})
            elif matches_any(RE_UNATTRIBUTED, sentence):
                out["unattributed_sourcing"].append({
                    "document_url": url, "location": f"paragraph {index + 1}",
                    "sentence": sentence})

    def _primary(self, url, cls, page, res):
        primary = {"org": cls["organisation"], "type": cls["type"], "title": None,
                   "date_published": None, "url": url, "access": "undisclosed",
                   "access_evidence": [], "stated_confidence": "not stated",
                   "confidence_evidence": [], "hedges": [], "interest": cls["interest"],
                   "resolution": "linked", "null_reasons": {}}
        if cls["notes"].lower().startswith("press release wire"):
            primary["org_note"] = ("press release wire: the issuing organisation named in the "
                                   "release is the source, not the wire")
        if page is None:
            reason = res.error or "document not read"
            primary["resolution"] = "linked_not_read"
            primary["null_reasons"] = {"title": reason, "date_published": reason}
            primary["access_note"] = "undisclosed because the document was not read"
            return primary
        primary["title"] = page.title
        primary["date_published"] = page.date_published
        if not primary["title"]:
            primary["null_reasons"]["title"] = "no title in page metadata"
        if not primary["date_published"]:
            primary["null_reasons"]["date_published"] = "no publication date in page metadata"
        counts = {}
        for index, sentence in page.sentences():
            location = f"paragraph {index + 1}"
            if matches_any(RE_CONFIDENCE, sentence):
                primary["confidence_evidence"].append({"sentence": sentence, "location": location})
            elif matches_any(RE_HEDGE, sentence) and len(primary["hedges"]) < 25:
                primary["hedges"].append({"sentence": sentence, "location": location})
            for name, patterns in RE_ACCESS:
                if matches_affirmed(patterns, sentence):
                    counts[name] = counts.get(name, 0) + 1
                    if len(primary["access_evidence"]) < 25:
                        primary["access_evidence"].append(
                            {"access": name, "sentence": sentence, "location": location})
        allowed = ACCESS_BY_TYPE.get(cls["type"], set())
        counts = {k: v for k, v in counts.items() if k in allowed}
        if counts:
            order = [name for name, _ in ACCESS_PATTERNS]
            primary["access"] = max(order, key=lambda n: (counts.get(n, 0), -order.index(n)))
        if primary["confidence_evidence"]:
            primary["stated_confidence"] = primary["confidence_evidence"][0]["sentence"]
        return primary

    def _summary(self, out):
        chain = out["chain"]
        if not chain:
            return "Nothing could be fetched."
        first = chain[0]
        name = first["org"] or urllib.parse.urlsplit(first["url"]).hostname
        primary = out["primary_source"]
        if out["status"] != "resolved":
            reasons = "; ".join(dict.fromkeys(
                u["reason"] for u in out["unresolved"]
                if u["field"] in ("primary_source", "chain")))
            return (f"This {name} page could not be traced to an originating source: "
                    f"{reasons}. Provenance is unresolved: access level untraced, claim "
                    "support tentative at best.")
        if len(chain) == 1:
            lead = f"This is a primary document from {primary['org']} ({primary['type']})"
        else:
            via = [h["org"] or urllib.parse.urlsplit(h["url"]).hostname for h in chain[1:-1]]
            route = "".join(f" via {v}'s coverage" for v in via)
            lead = (f"This {name} article{route} traces to a document from {primary['org']}"
                    f" ({primary['type']})")
        title = f", \"{primary['title']}\"" if primary.get("title") else ""
        date = (f", published {primary['date_published']}" if primary.get("date_published")
                else ", publication date not found")
        extra = ""
        if out["other_primaries"]:
            orgs = ", ".join(dict.fromkeys(o["org"] for o in out["other_primaries"]))
            extra = f" The article also links {orgs}; independence not established."
        if primary.get("date_check"):
            extra += " The primary's date could not be checked against the article's."
        if primary.get("resolution") == "linked_not_read":
            return (f"{lead}{date}. The document was located but not read "
                    f"({primary['null_reasons'].get('title')}): access and confidence "
                    f"unknown.{extra}")
        access = (f"Its text indicates access by {primary['access'].replace('_', ' ')}"
                  if primary["access"] != "undisclosed" else "It does not state how it knows")
        conf = ("it states its confidence" if primary["stated_confidence"] != "not stated"
                else "no confidence statement was found")
        return f"{lead}{title}{date}. {access}; {conf}.{extra}"


# --- CLI -------------------------------------------------------------------------------

def load_fixtures(path):
    if not path:
        return {}
    base = Path(path).resolve().parent
    mapping = json.loads(Path(path).read_text())
    out = {}
    for url, target in mapping.items():
        out[url] = target if isinstance(target, dict) else str((base / target).resolve())
    return out


def cmd_classify(args, table):
    print(json.dumps([dict(url=u, **table.lookup(u)) for u in args.urls], indent=2))
    return 0


def cmd_text(args, table):
    fetcher = Fetcher(table, args.cache_dir, not args.no_cache, load_fixtures(args.fixtures))
    res = fetcher.fetch(args.url)
    if not res.ok:
        print(json.dumps({"url": args.url, "status": res.status, "reason": res.error}, indent=2))
        return 2
    page = Page(res.final_url, res.body)
    path = fetcher.text_path(args.url)
    path.parent.mkdir(parents=True, exist_ok=True)
    path.write_text(page.text(), encoding="utf-8")
    print(json.dumps({"url": args.url, "status": "ok", "title": page.title,
                      "date_published": page.date_published, "text_file": str(path),
                      "paragraphs": len(page.paragraphs),
                      "note": "full text is local only; do not upload or commit it"}, indent=2))
    return 0


def cmd_resolve(args, table):
    urls = list(dict.fromkeys(args.urls))
    if len(urls) > MAX_URLS or args.max_hops > MAX_HOPS:
        raise Refused(
            f"Refused: this script resolves at most {MAX_URLS} URLs and {MAX_HOPS} hops per "
            "invocation. Bulk provenance is served by the Liberty91 API, where chains are "
            "resolved once at ingest (see /lookup-liberty91).")
    today = dt.date.fromisoformat(args.today) if args.today else dt.date.today()
    fetcher = Fetcher(table, args.cache_dir, not args.no_cache, load_fixtures(args.fixtures))
    resolver = Resolver(table, fetcher, max(1, args.max_hops), today)
    with ThreadPoolExecutor(max_workers=MAX_HOSTS_IN_FLIGHT) as pool:
        results = list(pool.map(resolver.resolve, urls))
    print(json.dumps({
        "tool": "resolve_provenance", "version": "2.0",
        "resolved_at": today.isoformat(), "max_hops": resolver.max_hops,
        "source_types_rows": len(table.rows),
        "results": results, "fetch_log": fetcher.log,
        "note": "fidelity is null on resolved hops: it is a model judgement made by "
                "/quality-of-information-check, not by this script",
    }, indent=2, ensure_ascii=False))
    return 0 if all(r["status"] == "resolved" for r in results) else 2


def main(argv=None):
    parser = argparse.ArgumentParser(description="Resolve provenance chains deterministically.")
    sub = parser.add_subparsers(dest="command", required=True)

    def common(p):
        p.add_argument("--no-cache", action="store_true", help="do not read or write the local cache")
        p.add_argument("--cache-dir", help="default ~/.cache/cti-skills/provenance or $CTI_SKILLS_CACHE")
        p.add_argument("--fixtures", help="JSON map of URL to local HTML file; disables the network")

    p = sub.add_parser("resolve", help="resolve one or more URLs to their primary")
    p.add_argument("urls", nargs="+")
    p.add_argument("--max-hops", type=int, default=MAX_HOPS,
                   help="1 = article to primary; 2 = article to article to primary (default 2)")
    p.add_argument("--today", help="override the assessment date (YYYY-MM-DD), for tests")
    common(p)

    p = sub.add_parser("classify", help="classify URLs against source-types.md; no network")
    p.add_argument("urls", nargs="+")

    p = sub.add_parser("text", help="extract article text to the local cache")
    p.add_argument("url")
    common(p)

    args = parser.parse_args(argv)
    try:
        table = SourceTable()
    except OSError as e:
        print(f"ERROR: cannot read {SOURCE_TYPES}: {e}", file=sys.stderr)
        return 1
    try:
        return {"resolve": cmd_resolve, "classify": cmd_classify, "text": cmd_text}[args.command](args, table)
    except Refused as e:
        print(str(e), file=sys.stderr)
        return 3


if __name__ == "__main__":
    sys.exit(main())
rea-analysis-optimizationmorluto/reaOptimize REA providers and shared analysis workflows for agent context, latency, CPU, and memory while preserving correctness and Evidence. Use for analysis bottlenecks and cache or query design; CI/test-suite optimization belongs to rea-ci-tests-audit.Stars: 472kUpdated Claude CodeCodexVerifiedInstall
rea-ci-tests-auditmorluto/reaAudit REA CI and test suites for correctness, fixture drift, false passes, and measured performance; use for suite or workflow investigations and efficiency changes, rather than ordinary feature testing or a single test failure.Stars: 472kUpdated Claude CodeCodexVerifiedInstall
rea-tool-designmorluto/reaAudit, investigate, design, or change REA investigation tools, CLI/MCP contracts, provider capabilities, and Evidence semantics.Stars: 472kUpdated Claude CodeCodexVerifiedInstall
ponytail-auditDietrichGebert/ponytailQuality audit of the whole repo: bugs, security, real load, missing tests, speed, and what to delete. Most important first.Stars: 194kUpdated Claude CodeCodexVerifiedInstall