Skip to content

alexgreensh/repo-forensics / skills/repo-forensics

repo-forensics

Flagged

Security forensics for git repos, AI skills, and MCP servers. Audits dependencies, detects prompt injection, credential theft, runtime dynamism, manifest drift, known CVEs, CISA KEV (actively exploited) vulns, and 2026 attack patterns. Not for fixing vulnerabilities or pentesting.

FileKindSize
.gitignoreother193 B
SKILL.mdskill md43.4 KB
checksums.jsonconfig10.8 KB
checksums.json.sigother64 B
data/README.mdmarkdown1.3 KB
data/compromised_versions.jsonconfig188.3 KB
data/rule_ids.csvother35.3 KB
data/rulepacks/.gitkeepother221 B
data/rulepacks/dead_anchors.jsonconfig9.0 KB
data/rulepacks/mcp_security.jsonconfig31.2 KB
data/rulepacks/runtime_dynamism.jsonconfig46.7 KB
data/rulepacks/sast.jsonconfig107.6 KB
data/rulepacks/secrets.jsonconfig30.4 KB
data/rulepacks/shared.jsonconfig4.4 KB
data/rulepacks/skill_threats.jsonconfig95.4 KB
data/runtime_floors.jsonconfig2.3 KB
data/yara/cryptominers.yarother1.8 KB
data/yara/hacktools.yarother2.2 KB
data/yara/malware.yarother2.2 KB
data/yara/manifest.jsonconfig8.0 KB
data/yara/webshells.yarother2.5 KB
references/cursor-hook-contract.mdmarkdown5.6 KB
references/kernel_privesc_sandbox_pack.mdmarkdown5.0 KB
references/mcp-attack-patterns.mdmarkdown8.7 KB
references/research_sources.mdmarkdown8.0 KB
references/threat_patterns.mdmarkdown20.3 KB
scripts/_context_gate.pyscript21.7 KB
scripts/_ed25519.pyscript5.1 KB
scripts/_pyc_unmarshal.pyscript17.1 KB
scripts/_shared_patterns.pyscript4.8 KB
scripts/adjudication.pyscript16.4 KB
scripts/adjudication_bridge.pyscript1.0 KB
scripts/aggregate_json.pyscript69.2 KB
scripts/auto_scan.pyexecutablescript152.1 KB
scripts/corpus_sync.pyscript10.0 KB
scripts/cursor_pre_scan.pyscript22.5 KB
scripts/dast_sandbox.sbother1.6 KB
scripts/dead_anchors_extract.pyscript14.8 KB
scripts/dead_anchors_probe.pyscript27.1 KB
scripts/forensics_core.pyscript176.5 KB
scripts/gen_rule_ids.pyscript8.6 KB
scripts/hook_adapter.pyscript19.4 KB
scripts/ioc_manager.pyscript49.6 KB
scripts/parse_pnpm_lock.pyscript7.9 KB
scripts/pre_scan.pyscript143.5 KB
scripts/pypi_pins.pyscript8.1 KB
scripts/refresh_controller.pyexecutablescript48.4 KB
scripts/refresh_threat_dbs.pyexecutablescript19.4 KB
scripts/rule_loader.pyscript44.8 KB
scripts/rulepack_feed.pyscript26.4 KB
scripts/run_forensics.shexecutablescript20.3 KB
scripts/sarif_export.pyscript18.6 KB
scripts/scan_agent_skills.pyscript33.2 KB
scripts/scan_archive.pyscript29.1 KB
scripts/scan_ast.pyscript44.9 KB
scripts/scan_binary.pyscript10.9 KB
scripts/scan_bytecode.pyscript28.2 KB
scripts/scan_dast.pyscript28.8 KB
scripts/scan_dataflow.pyscript15.4 KB
scripts/scan_dead_anchors.pyscript15.9 KB
scripts/scan_decode.pyscript37.5 KB
scripts/scan_dependencies.pyscript124.5 KB
scripts/scan_devcontainer.pyscript8.5 KB
scripts/scan_entropy.pyscript15.6 KB
scripts/scan_entrypoint.pyscript34.3 KB
scripts/scan_git_config.pyscript96.3 KB
scripts/scan_git_forensics.pyscript24.1 KB
scripts/scan_history.pyscript14.5 KB
scripts/scan_infra.pyscript54.3 KB
scripts/scan_integrity.pyscript41.0 KB
scripts/scan_lifecycle.pyscript65.1 KB
scripts/scan_manifest_drift.pyscript47.0 KB
scripts/scan_mcp_security.pyscript42.3 KB
scripts/scan_oversize.pyscript11.7 KB
scripts/scan_post_incident.pyscript36.9 KB
scripts/scan_provenance.pyscript19.4 KB
scripts/scan_runtime_dynamism.pyscript23.5 KB
scripts/scan_runtime_floors.pyscript33.5 KB
scripts/scan_sast.pyscript59.1 KB
scripts/scan_secrets.pyscript11.9 KB
scripts/scan_skill_threats.pyscript102.4 KB
scripts/scan_splitstream.pyscript29.5 KB
scripts/scan_yara.pyscript19.7 KB
scripts/session_scan.pyscript54.1 KB
scripts/suppressions.pyscript15.4 KB
scripts/validate_manifests.pyscript13.7 KB
scripts/verify_install.pyscript26.2 KB
scripts/vuln_feed.pyscript31.1 KB
tests/conftest.pyscript13.4 KB
tests/corpus/DEAD_ANCHORS_CORPUS.mdmarkdown2.3 KB
tests/corpus/budgets.jsonconfig5.6 KB
tests/cursor_helpers.pyscript12.6 KB
tests/fixtures/legacy_main_blocks.jsonconfig10.3 KB
tests/fixtures/powershell_verdict_corpus.jsonconfig114.9 KB
tests/golden/parity_mcp_security.jsonconfig17.5 KB
tests/golden/parity_runtime_dynamism.jsonconfig31.7 KB
tests/golden/parity_sast.jsonconfig59.8 KB
tests/golden/parity_secrets.jsonconfig18.8 KB
tests/golden/parity_skill_threats.jsonconfig48.8 KB
tests/parity_corpus.pyscript6.3 KB
tests/powershell_hook.pyscript8.2 KB
tests/report_builders.pyscript5.0 KB
tests/schemas/sarif-schema-2.1.0.jsonconfig109.1 KB
tests/shell_compat.pyscript2.2 KB
tests/test_adjudication_bridge.pyscript832 B
tests/test_adjudication_containment.pyscript1.4 KB
tests/test_adjudication_output.pyscript20.0 KB
tests/test_adjudication_suppressions.pyscript18.7 KB
tests/test_advisory_adjudication.pyscript1.3 KB
tests/test_agent_relay_rules.pyscript7.8 KB
tests/test_aggregate_json.pyscript10.0 KB
tests/test_auto_scan_patterns.pyscript10.5 KB
tests/test_benign_corpus.pyscript16.4 KB
tests/test_benign_new_scanners.pyscript6.1 KB
tests/test_build_output_skip.pyscript10.6 KB
tests/test_capability_ledger.pyscript18.8 KB
tests/test_codex_install.pyscript10.0 KB
tests/test_codex_marketplace.pyscript3.3 KB
tests/test_comment_false_positives.pyscript2.2 KB
tests/test_composer_dependencies.pyscript4.0 KB
tests/test_compromised_versions.pyscript51.2 KB
tests/test_context_gate.pyscript32.4 KB
tests/test_coverage_status.pyscript7.6 KB
tests/test_css_compounding_shrink.pyscript20.8 KB
tests/test_cursor_adapter.pyscript7.8 KB
tests/test_cursor_benign.pyscript3.4 KB
tests/test_cursor_degrade.pyscript12.7 KB
tests/test_cursor_install.pyscript17.1 KB
tests/test_cursor_latency.pyscript7.2 KB
tests/test_cursor_parity.pyscript5.8 KB
tests/test_cursor_redteam_fn.pyscript4.9 KB
tests/test_cursor_tamper_modify.pyscript14.2 KB
tests/test_cursor_wiring.pyscript32.3 KB
tests/test_d1_yara_correlation.pyscript9.2 KB
tests/test_dead_anchors_extract.pyscript7.6 KB
tests/test_dead_anchors_probe.pyscript19.7 KB
tests/test_evidence_engine_parity.pyscript8.1 KB
tests/test_evidence_model.pyscript11.4 KB
tests/test_forensics_core.pyscript79.2 KB
tests/test_fp_hardening_r3.pyscript9.7 KB
tests/test_fp_hardening_r4.pyscript18.8 KB
tests/test_fp_regression_pairs.pyscript62.5 KB
tests/test_fp_structural.pyscript21.2 KB
tests/test_freshness.pyscript6.2 KB
tests/test_frontmatter_validity.pyscript7.0 KB
tests/test_gpg_rce_hotfix.pyscript4.8 KB
tests/test_hook_inert_data.pyscript6.1 KB
tests/test_hook_scripts.pyscript14.0 KB
tests/test_inspection_cpu_budget.pyscript11.6 KB
tests/test_ioc_manager.pyscript55.4 KB
tests/test_kernel_privesc_rules.pyscript27.8 KB
tests/test_legit_precision_regressions.pyscript6.8 KB
tests/test_lethal_trifecta.pyscript20.7 KB
tests/test_manifest_signature.pyscript1.5 KB
tests/test_manifest_validity.pyscript2.9 KB
tests/test_monotonic_trust_e2e.pyscript4.4 KB
tests/test_non_breaking_contract.pyscript22.7 KB
tests/test_openclaw_install.pyscript6.6 KB
tests/test_pack_parity.pyscript24.8 KB
tests/test_parse_pnpm_lock.pyscript10.7 KB
tests/test_plugin_checkout_provenance.pyscript11.1 KB
tests/test_powershell_hook_wire.pyscript10.6 KB
tests/test_powershell_verdict_corpus.pyscript6.2 KB
tests/test_pr44_hardening.pyscript6.5 KB
tests/test_pre_scan.pyscript19.6 KB
tests/test_pre_scan_composer.pyscript7.4 KB
tests/test_precondition_fields.pyscript4.6 KB
tests/test_provenance_offline.pyscript2.5 KB
tests/test_published_rulepack_feed_current.pyscript1.4 KB
tests/test_py38_compat.pyscript3.3 KB
tests/test_pypi_pinned_versions.pyscript19.3 KB
tests/test_pyproject_string_forms.pyscript2.2 KB
tests/test_redteam_fn_regression.pyscript32.5 KB
tests/test_reflective_fetch_execute.pyscript23.5 KB
tests/test_reflective_review_fixes.pyscript17.8 KB
tests/test_refresh_controller.pyscript19.9 KB
tests/test_refresh_rulepacks.pyscript12.4 KB
tests/test_review_gate_regressions.pyscript9.3 KB
tests/test_rule_loader.pyscript23.2 KB
tests/test_rulepack_feed.pyscript38.1 KB
tests/test_run_forensics_json.pyscript1012 B
tests/test_run_scanner_fail_loud.pyscript6.5 KB
tests/test_runtime_floors.pyscript19.8 KB
tests/test_sandbox_probing_rules.pyscript9.6 KB
tests/test_sarif_export.pyscript19.9 KB
tests/test_scan_agent_skills.pyscript31.3 KB
tests/test_scan_archive.pyscript11.6 KB
tests/test_scan_ast.pyscript12.8 KB
tests/test_scan_binary.pyscript15.3 KB
tests/test_scan_bytecode.pyscript21.5 KB
tests/test_scan_dast.pyscript32.1 KB
tests/test_scan_dataflow.pyscript7.1 KB
tests/test_scan_dead_anchors.pyscript11.5 KB
tests/test_scan_decode.pyscript28.0 KB
tests/test_scan_dependencies.pyscript37.8 KB
tests/test_scan_devcontainer.pyscript8.5 KB
tests/test_scan_entropy.pyscript11.2 KB
tests/test_scan_entrypoint.pyscript19.5 KB
tests/test_scan_git_config.pyscript58.5 KB
tests/test_scan_git_config_review_fixes.pyscript33.3 KB

scripts/dead_anchors_probe.py

View on GitHub
#!/usr/bin/env python3
"""
dead_anchors_probe.py - Network claimability layer for the dead-anchor scanner
(U2).

Turns an extracted Anchor into a CC / LO / NC verdict:
  CC = CONFIRMED-CLAIMABLE (404 / NXDOMAIN / RDAP-404 / provider fingerprint) -> the
       one signal that emits a Finding downstream.
  LO = LIVE-AND-OWNED (200 / registered) -> silent (normal state of the world).
  NC = COULDN'T-CHECK (403 / 429 / 5xx / timeout / offline / over-budget) -> silent.

Loop-safety & race-safety contract (spec §10), all STRUCTURAL not incidental:
  * Single attempt per probe. There is NO retry loop anywhere in this module.
  * HTTPError caught FIRST, branched on .code (KTD-3): 404 => CC (signal),
    403/429 => NC + trips the per-host circuit breaker, 5xx/other => NC. A 404
    NEVER collapses into the generic "network failed => NC" branch (inverts the
    vuln_feed.fetch_npm/pypi_freshness anti-pattern).
  * Per-host circuit breaker: first 403/429 from a host trips it; every later
    probe to that host short-circuits to NC with zero further calls, no reset.
  * One shared wall-clock deadline + a GH call cap + a total-probe ceiling; all
    checked BEFORE any socket is touched (network last).
  * 24h atomic result cache, reusing vuln_feed._atomic_write / _load_cache
    directly (temp+os.replace+0o600, no lock, tolerant reads). Only CC/LO are
    cached; NC (transient) always re-probes next scan.
  * offline=True is the cheapest path: instant NC before cache/deadline/budget.

All HTTPS goes through vuln_feed._https_fetch (no 4th hand-rolled urlopen). DNS
for cloud subdomains uses socket.gethostbyname (no HTTP equivalent), wrapped in
the same never-raise discipline.

Created by Alex Greenshpun
"""

import os
import re
import sys
import json
import time
import stat
import socket
import hashlib
import tempfile
import ipaddress
import threading
import http.client
import urllib.error
import urllib.parse
import urllib.request

_SCRIPTS_DIR = os.path.dirname(os.path.abspath(__file__))
if _SCRIPTS_DIR not in sys.path:
    sys.path.insert(0, _SCRIPTS_DIR)

import vuln_feed
import forensics_core as core
from dead_anchors_extract import normalize_npm_name, normalize_pypi_name

# ---- Tunables (read at construction so tests can monkeypatch) --------------
NETWORK_DEADLINE_SEC = 12.0        # whole-pass wall-clock budget
GITHUB_CALL_CAP = 20               # unauthenticated GH calls/scan
GITHUB_CALL_CAP_TOKEN = 40         # raised (not authenticated) when a token exists
TOTAL_PROBE_CEILING = 50           # across ALL anchor types/scan
RESULT_CACHE_TTL_HOURS = 24

_HTTP_MAX_BYTES = 512 * 1024       # cap on any probe response body
_CLOUD_BODY_MAX_BYTES = 128 * 1024

_HOST_GITHUB = "api.github.com"
_HOST_NPM = "registry.npmjs.org"
_HOST_PYPI = "pypi.org"
_HOST_RDAP = "rdap.org"

# Input-validation charsets (defense-in-depth before a value hits a URL).
_OWNER_RE = re.compile(r"^[A-Za-z0-9][A-Za-z0-9._-]{0,100}$")
_PKG_RE = re.compile(r"^@?[A-Za-z0-9][A-Za-z0-9@/._-]{0,120}$")
_DOMAIN_RE = re.compile(r"^[a-z0-9][a-z0-9.-]{0,190}$")


# ---------------------------------------------------------------------------
# SSRF / redirect hardening (security-sentinel F1/F4, code-review F1).
#
# Python's DEFAULT urllib opener transparently follows up to 10 redirects, each
# a fresh request to a SERVER-CONTROLLED Location that bypasses the per-host
# breaker, the probe budget, the whole-pass deadline AND the https-only check
# (https->http downgrade + pivot to internal / link-local / metadata IPs are all
# allowed). Every dead_anchors probe therefore uses an EXPLICIT opener:
#   * fixed-host + attacker-host probes (GitHub/npm/PyPI/cloud) -> NO redirects.
#     A 3xx surfaces as its own HTTPError and is classified COULDN'T-CHECK; the
#     Location is never auto-fetched.
#   * rdap.org -> a BOUNDED, re-validated redirect (its whole purpose is to
#     bootstrap-redirect to the correct registry RDAP server): <=3 hops, each
#     https-only and each rejected if the target host resolves into a private /
#     loopback / link-local / metadata range.
# ---------------------------------------------------------------------------

def _is_internal_ip(ip):
    """True for any address a forensics probe must never connect to."""
    try:
        addr = ipaddress.ip_address(ip)
    except (ValueError, TypeError):
        return True  # unparseable => refuse (fail closed)
    return (addr.is_private or addr.is_loopback or addr.is_link_local
            or addr.is_multicast or addr.is_reserved or addr.is_unspecified)


def _hostname_is_internal(host):
    """Resolve host (A+AAAA) and return True if ANY address is internal, or if it
    cannot be resolved/validated (fail closed — block the redirect hop)."""
    try:
        infos = socket.getaddrinfo(host, 443, proto=socket.IPPROTO_TCP)
    except Exception:
        return True
    checked = False
    for info in infos:
        sockaddr = info[4] if len(info) > 4 else None
        if sockaddr:
            checked = True
            if _is_internal_ip(sockaddr[0]):
                return True
    return not checked


class _NoRedirectHandler(urllib.request.HTTPRedirectHandler):
    """Refuse ALL redirects. Returning None makes urllib raise the 3xx as an
    HTTPError instead of following the Location header."""

    def redirect_request(self, req, fp, code, msg, headers, newurl):
        return None


class _BoundedRedirectHandler(urllib.request.HTTPRedirectHandler):
    """Follow at most a few redirects, each re-validated: https-only, and the
    target host must not resolve into an internal range."""

    max_redirections = 3

    def redirect_request(self, req, fp, code, msg, headers, newurl):
        try:
            parts = urllib.parse.urlsplit(newurl)
        except Exception:
            return None
        if parts.scheme != "https":
            return None
        host = parts.hostname
        if not host or _hostname_is_internal(host):
            return None
        return super().redirect_request(req, fp, code, msg, headers, newurl)


_NO_REDIRECT_OPENER = None
_BOUNDED_REDIRECT_OPENER = None


def _no_redirect_opener():
    global _NO_REDIRECT_OPENER
    if _NO_REDIRECT_OPENER is None:
        _NO_REDIRECT_OPENER = urllib.request.build_opener(_NoRedirectHandler)
    return _NO_REDIRECT_OPENER


def _bounded_redirect_opener():
    global _BOUNDED_REDIRECT_OPENER
    if _BOUNDED_REDIRECT_OPENER is None:
        _BOUNDED_REDIRECT_OPENER = urllib.request.build_opener(_BoundedRedirectHandler)
    return _BOUNDED_REDIRECT_OPENER


class _DirectOpener:
    """Test seam: an opener whose .open() routes back through
    vuln_feed.urllib.request.urlopen so the suite's single mock chokepoint still
    intercepts every dead_anchors probe. Production always uses the real
    no-redirect / bounded-redirect openers above."""

    def open(self, req, timeout=None):
        return vuln_feed.urllib.request.urlopen(req, timeout=timeout)


class _Deadline:
    """Single shared wall-clock budget for the whole claimability pass (mirrors
    scan_provenance._Deadline). NETWORK_DEADLINE_SEC read at construction so a
    test can monkeypatch it."""

    def __init__(self):
        self._end = time.monotonic() + NETWORK_DEADLINE_SEC

    def remaining(self):
        return max(0.0, self._end - time.monotonic())

    def expired(self):
        return self.remaining() <= 0.0


class _ProbeBudget:
    """GH-specific cap + total-probe ceiling. Consumed only when a probe is
    actually about to issue a request."""

    def __init__(self, github_cap, total_ceiling):
        self.gh_remaining = github_cap
        self.total_remaining = total_ceiling

    def take(self, github=False):
        if self.total_remaining <= 0:
            return False
        if github and self.gh_remaining <= 0:
            return False
        self.total_remaining -= 1
        if github:
            self.gh_remaining -= 1
        return True


class _CircuitBreaker:
    """Per-scan, per-host trip state. First 403/429 from a host trips it; never
    resets within the scan (no backoff, no Retry-After honoring)."""

    def __init__(self):
        self._tripped = set()

    def trip(self, host):
        self._tripped.add(host)

    def is_tripped(self, host):
        return host in self._tripped


class ProbeContext:
    """Bundles the per-scan state threaded through every probe. All state is
    LOCAL to one scan_repo() call — no module-level mutable globals (spec §10
    'no shared mutable state across scanners')."""

    def __init__(self, offline=False, cache_dir=None, github_token=None):
        self.offline = offline
        self.deadline = _Deadline()
        token = github_token if github_token is not None else os.environ.get("GITHUB_TOKEN")
        self.has_token = bool(token)
        # The token is NEVER attached to a request (no Authorization header is
        # added anywhere in the fetch path), so authenticated calls are not
        # actually made — a token must therefore NOT raise the call budget.
        # Inflating it only made the scanner hammer GitHub's real 60/hr limit,
        # trip the circuit breaker sooner and silently under-detect
        # (security-sentinel F7). Keep the cap fixed until real auth is wired.
        self.budget = _ProbeBudget(GITHUB_CALL_CAP, TOTAL_PROBE_CEILING)
        self.breaker = _CircuitBreaker()
        self.cache_dir = cache_dir if cache_dir is not None else _default_cache_dir()

    # -- 24h result cache (only CC/LO cached; NC always re-probes) ------------

    def _cache_path(self, key):
        digest = hashlib.sha256(key.encode("utf-8")).hexdigest()
        return os.path.join(self.cache_dir, f"{digest}.json")

    def cache_get(self, key):
        """Return a cached (verdict, extra) tuple within TTL, else None.
        Tolerant: any read error / malformed / wrong-shape entry => cache-miss,
        never a crash (KTD-14). Refuses to read through a symlinked cache dir or
        entry (verdict-poisoning backstop, security-sentinel F6)."""
        try:
            path = self._cache_path(key)
            if os.path.islink(self.cache_dir) or os.path.islink(path):
                return None
            data = vuln_feed._load_cache(path, RESULT_CACHE_TTL_HOURS)
        except Exception:
            return None
        if not isinstance(data, dict):
            return None
        verdict = data.get("verdict")
        if verdict not in ("CC", "LO"):
            return None
        extra = data.get("extra")
        return verdict, (extra if isinstance(extra, dict) else None)

    def cache_put(self, key, verdict, extra=None):
        if verdict not in ("CC", "LO"):
            return  # never cache a transient NC
        try:
            path = self._cache_path(key)
            if os.path.islink(self.cache_dir) or os.path.islink(path):
                return  # never write through a symlinked cache dir/entry
        except OSError:
            return
        payload = {"_cached_at": time.time(), "verdict": verdict,
                   "extra": extra if isinstance(extra, dict) else {}}
        try:
            vuln_feed._atomic_write(path, payload)
        except OSError:
            pass  # cache write failure is non-fatal


def _default_cache_dir():
    """Never raise: in a UID-less container `~` may not expand
    (pwd.getpwuid -> KeyError), which must degrade to a temp dir, not crash the
    scan (python-redos F4)."""
    try:
        home = os.path.expanduser("~")
        if not home or home == "~":
            raise ValueError("no home directory")
        return os.path.join(home, ".cache", "repo-forensics", "dead-anchors")
    except Exception:
        return os.path.join(tempfile.gettempdir(), "repo-forensics-dead-anchors")


ProbeResult = None  # (kept as documentation; probes return (verdict, extra))


# ---------------------------------------------------------------------------
# Core HTTPS wrapper — the single most load-bearing correctness property.
# HTTPError caught BEFORE URLError, branched on .code. Single attempt.
# ---------------------------------------------------------------------------

def _call_timeout(ctx):
    """Per-call socket timeout = min(network timeout, remaining pass deadline) so
    one hung probe can never overrun the whole-pass budget by a full timeout
    (code-review F2, integration F4)."""
    remaining = ctx.deadline.remaining()
    base = getattr(vuln_feed, "NETWORK_TIMEOUT_SEC", 10)
    if remaining <= 0:
        return 0.001
    return min(base, remaining)


def _probe_https(url, host, ctx, max_bytes=_HTTP_MAX_BYTES, opener=None):
    """Issue ONE GET via the hardened vuln_feed._https_fetch, with NO redirect
    following (opener defaults to the no-redirect opener). Returns
    (verdict, raw_bytes_or_None). On 403/429 trips the host breaker. A 3xx
    (redirect refused) is classified NC — never auto-fetched. Never raises."""
    if opener is None:
        opener = _no_redirect_opener()
    try:
        raw = vuln_feed._https_fetch(url, max_bytes, opener=opener,
                                     timeout=_call_timeout(ctx))
        return "LO", raw
    except urllib.error.HTTPError as e:  # MUST precede URLError (subclass)
        code = getattr(e, "code", None)
        if code == 404:
            return "CC", None
        if code in (403, 429):
            ctx.breaker.trip(host)
            return "NC", None
        return "NC", None  # incl. 3xx (redirect refused) => couldn't-check
    except (urllib.error.URLError, OSError, ValueError, http.client.HTTPException):
        return "NC", None


def _pre_network_ok(ctx, host, github=False):
    """Shared cheap gate run before any request: deadline -> breaker -> budget.
    Returns True iff a request may be issued. 'network last'."""
    if ctx.deadline.expired():
        return False
    if ctx.breaker.is_tripped(host):
        return False
    if not ctx.budget.take(github=github):
        return False
    return True


def _parse_json(raw):
    if not raw:
        return None
    try:
        return json.loads(raw.decode("utf-8"))
    except (ValueError, UnicodeDecodeError):
        return None


# ---------------------------------------------------------------------------
# Per-anchor-type probes. Order in each: offline -> cache -> pre-network -> fetch.
# ---------------------------------------------------------------------------

def probe_github_user(owner, ctx):
    """GET api.github.com/users/{owner}. On LO returns the parsed JSON (for
    DA-10 owner-trust enrichment)."""
    if ctx.offline:
        return "NC", None
    if not _OWNER_RE.match(owner or ""):
        return "NC", None
    key = f"gh_user::{owner.lower()}"
    cached = ctx.cache_get(key)
    if cached is not None:
        return cached
    if not _pre_network_ok(ctx, _HOST_GITHUB, github=True):
        return "NC", None
    url = f"https://api.github.com/users/{urllib.parse.quote(owner, safe='')}"
    verdict, raw = _probe_https(url, _HOST_GITHUB, ctx)
    meta = _github_owner_meta(_parse_json(raw)) if verdict == "LO" else None
    if verdict in ("CC", "LO"):
        ctx.cache_put(key, verdict, meta)
    return verdict, meta


def probe_github_repo(owner, repo, ctx):
    """GET api.github.com/repos/{owner}/{repo}. Only meaningful when the owner
    probe returned LO (caller enforces)."""
    if ctx.offline:
        return "NC", None
    if not (_OWNER_RE.match(owner or "") and _OWNER_RE.match(repo or "")):
        return "NC", None
    key = f"gh_repo::{owner.lower()}/{repo.lower()}"
    cached = ctx.cache_get(key)
    if cached is not None:
        return cached
    if not _pre_network_ok(ctx, _HOST_GITHUB, github=True):
        return "NC", None
    url = (f"https://api.github.com/repos/{urllib.parse.quote(owner, safe='')}/"
           f"{urllib.parse.quote(repo, safe='')}")
    verdict, _raw = _probe_https(url, _HOST_GITHUB, ctx)
    if verdict in ("CC", "LO"):
        ctx.cache_put(key, verdict, None)
    return verdict, None


def probe_npm(name, ctx):
    if ctx.offline:
        return "NC", None
    if not _PKG_RE.match(name or "") or ".." in name:
        return "NC", None
    # npm names are case-insensitive & lowercase: probe (and cache) the canonical
    # form so 'npm install React' resolves to the live 'react', not a phantom
    # 404 (integration F2).
    probe_name = normalize_npm_name(name)
    if not probe_name:
        return "NC", None
    key = f"npm::{probe_name}"
    cached = ctx.cache_get(key)
    if cached is not None:
        return cached
    if not _pre_network_ok(ctx, _HOST_NPM):
        return "NC", None
    url = f"https://registry.npmjs.org/{urllib.parse.quote(probe_name, safe='@')}"
    verdict, _raw = _probe_https(url, _HOST_NPM, ctx)
    if verdict in ("CC", "LO"):
        ctx.cache_put(key, verdict, None)
    return verdict, None


def probe_pypi(name, ctx):
    if ctx.offline:
        return "NC", None
    if not _PKG_RE.match(name or "") or ".." in name:
        return "NC", None
    # PEP 503 canonicalization before probing: 'My_Package', 'my-package',
    # 'my.package' are one live distribution, not a phantom (integration F3).
    probe_name = normalize_pypi_name(name)
    if not probe_name:
        return "NC", None
    key = f"pypi::{probe_name}"
    cached = ctx.cache_get(key)
    if cached is not None:
        return cached
    if not _pre_network_ok(ctx, _HOST_PYPI):
        return "NC", None
    url = f"https://pypi.org/pypi/{urllib.parse.quote(probe_name, safe='')}/json"
    verdict, _raw = _probe_https(url, _HOST_PYPI, ctx)
    if verdict in ("CC", "LO"):
        ctx.cache_put(key, verdict, None)
    return verdict, None


def _rdap_probe(url, ctx):
    """rdap.org fetch with BOUNDED, re-validated redirects. Returns
    (verdict, extra). A 404 is CC only when it was delivered by a delegated
    registry RDAP server (i.e. a redirect off rdap.org actually happened —
    genuine 'domain does not exist'); a bare 404 straight from rdap.org means it
    could not bootstrap a server for that TLD (ccTLD gap / transient) and must
    NOT fire a false CRITICAL on a live domain (security-sentinel F2)."""
    opener = _bounded_redirect_opener()
    try:
        raw = vuln_feed._https_fetch(url, _HTTP_MAX_BYTES, opener=opener,
                                     timeout=_call_timeout(ctx))
        return "LO", _rdap_expiry_meta(_parse_json(raw))
    except urllib.error.HTTPError as e:
        code = getattr(e, "code", None)
        if code in (403, 429):
            ctx.breaker.trip(_HOST_RDAP)
            return "NC", None
        if code == 404:
            final = getattr(e, "url", "") or ""
            try:
                host = (urllib.parse.urlsplit(final).hostname or "").lower()
            except Exception:
                host = ""
            if host and host != _HOST_RDAP:
                return "CC", None      # registry itself said 'not found'
            return "NC", None          # rdap.org bootstrap gap => ambiguous
        return "NC", None
    except (urllib.error.URLError, OSError, ValueError, http.client.HTTPException):
        return "NC", None


def probe_domain_rdap(domain, ctx):
    """GET rdap.org/domain/{domain}. Registry 404 (after bootstrap redirect) =>
    CC (unregistered/expired). On LO, inspect the events array for a PAST
    expiration date (imminent-expiry flag)."""
    if ctx.offline:
        return "NC", None
    if not _DOMAIN_RE.match(domain or ""):
        return "NC", None
    key = f"rdap::{domain.lower()}"
    cached = ctx.cache_get(key)
    if cached is not None:
        return cached
    if not _pre_network_ok(ctx, _HOST_RDAP):
        return "NC", None
    url = f"https://rdap.org/domain/{urllib.parse.quote(domain, safe='')}"
    verdict, extra = _rdap_probe(url, ctx)
    if verdict in ("CC", "LO"):
        ctx.cache_put(key, verdict, extra)
    return verdict, extra


def _resolve_with_timeout(host, timeout):
    """Resolve host (A+AAAA via getaddrinfo, not IPv4-only gethostbyname) with a
    HARD wall-clock bound so a hung resolver can never overrun the pass deadline
    (integration F4). Returns (status, addrs):
      'ok'       -> addrs is the list of resolved IP strings
      'nxdomain' -> the name provably does not resolve (EAI_NONAME)
      'nc'       -> couldn't-check (timeout / temporary failure / other)
    An AAAA-only (IPv6-only) host resolves to 'ok', NOT a false 'nxdomain'
    (security-sentinel F3). A hung resolver returns 'nc', never 'nxdomain'."""
    result = {}

    def _run():
        try:
            infos = socket.getaddrinfo(host, 443, proto=socket.IPPROTO_TCP)
            result["addrs"] = [i[4][0] for i in infos
                               if i and len(i) > 4 and i[4]]
        except socket.gaierror as e:
            if getattr(e, "errno", None) == socket.EAI_NONAME:
                result["nx"] = True
            else:
                result["err"] = True
        except (OSError, UnicodeError, ValueError):
            result["err"] = True

    t = threading.Thread(target=_run, daemon=True)
    t.start()
    t.join(timeout if timeout and timeout > 0 else None)
    if t.is_alive():
        return "nc", None            # resolver hung -> couldn't-check
    if result.get("addrs"):
        return "ok", result["addrs"]
    if result.get("nx"):
        return "nxdomain", None
    return "nc", None


def _cloud_http(url, suffix, ctx):
    """Fetch a cloud subdomain root with NO redirects, capturing the response
    body EVEN ON a non-200 status (a deleted-app error page is what we need to
    fingerprint — the generic _probe_https discards the 404 body). Returns
    (body_bytes, ok); ok is False on transport failure / 403 / 429 (=> NC)."""
    try:
        raw = vuln_feed._https_fetch(url, _CLOUD_BODY_MAX_BYTES,
                                     opener=_no_redirect_opener(),
                                     timeout=_call_timeout(ctx))
        return raw, True
    except urllib.error.HTTPError as e:
        code = getattr(e, "code", None)
        if code in (403, 429):
            ctx.breaker.trip(suffix)
            return None, False
        try:
            body = e.read(_CLOUD_BODY_MAX_BYTES + 1)
            if body and len(body) > _CLOUD_BODY_MAX_BYTES:
                body = body[:_CLOUD_BODY_MAX_BYTES]
        except Exception:
            body = b""
        return body or b"", True
    except (urllib.error.URLError, OSError, ValueError, http.client.HTTPException):
        return None, False


def probe_cloud_subdomain(subdomain, suffix, ctx, fingerprints=None):
    """Claimability for a free-tier cloud subdomain.

    The ONLY stand-alone high-confidence claimable signal is DNS NXDOMAIN. A
    resolving host is fetched and its body checked against STRICT provider
    deleted-app fingerprints; a bare 404/200 from a resolving host with no strict
    fingerprint stays LO (silent), never a false CRITICAL (error-soundness F2,
    integration F1). Biased hard toward silence over a false CRITICAL."""
    if ctx.offline:
        return "NC", None
    if not _DOMAIN_RE.match(subdomain or ""):
        return "NC", None
    key = f"cloud::{subdomain.lower()}"
    cached = ctx.cache_get(key)
    if cached is not None:
        return cached
    # DNS resolution (no HTTP equivalent). Counts against the total ceiling +
    # deadline, but is not a GH call. Breaker keyed on the registrable suffix.
    if ctx.deadline.expired() or ctx.breaker.is_tripped(suffix):
        return "NC", None
    if not ctx.budget.take(github=False):
        return "NC", None
    status, addrs = _resolve_with_timeout(subdomain, _call_timeout(ctx))
    if status == "nxdomain":
        ctx.cache_put(key, "CC", {"reason": "nxdomain"})
        return "CC", {"reason": "nxdomain"}
    if status != "ok":
        return "NC", None
    # Structural SSRF backstop, independent of the rulepack: never fetch a
    # subdomain that resolves into a private / loopback / link-local / metadata
    # range, whatever the suffix list says (security-sentinel F1/F4).
    if any(_is_internal_ip(a) for a in (addrs or [])):
        return "NC", None
    # Resolves & public: fetch and fingerprint. Second call — guard again.
    if not _pre_network_ok(ctx, suffix):
        return "NC", None
    body, ok = _cloud_http(f"https://{subdomain}", suffix, ctx)
    if not ok:
        return "NC", None
    fp = _fingerprint_match(body, suffix, fingerprints or {})
    if fp is not None:
        ctx.cache_put(key, "CC", {"reason": "fingerprint", "confidence": fp})
        return "CC", {"reason": "fingerprint", "confidence": fp}
    ctx.cache_put(key, "LO", None)
    return "LO", None


# ---- Enrichment helpers ----------------------------------------------------

def _github_owner_meta(data):
    """DA-10 owner-trust signals from the SAME users response (created_at,
    public_repos, bio). Never raises."""
    if not isinstance(data, dict):
        return None
    return {
        "created_at": data.get("created_at"),
        "public_repos": data.get("public_repos"),
        "bio": data.get("bio"),
    }


def _rdap_expiry_meta(data):
    """Return {'imminent_expiry': True} if the RDAP events array carries a past
    (already-elapsed) expiration date, else None."""
    if not isinstance(data, dict):
        return None
    events = data.get("events")
    if not isinstance(events, list):
        return None
    now = time.time()
    for ev in events:
        if not isinstance(ev, dict):
            continue
        if ev.get("eventAction") != "expiration":
            continue
        ts = _parse_iso8601(ev.get("eventDate"))
        if ts is not None and ts < now:
            return {"imminent_expiry": True}
    return None


def _parse_iso8601(s):
    if not isinstance(s, str) or not s:
        return None
    txt = s.strip().replace("Z", "+00:00")
    for fmt in ("%Y-%m-%dT%H:%M:%S%z", "%Y-%m-%dT%H:%M:%S.%f%z",
                "%Y-%m-%dT%H:%M:%S", "%Y-%m-%d"):
        try:
            import datetime
            dt = datetime.datetime.strptime(txt, fmt)
            if dt.tzinfo is None:
                dt = dt.replace(tzinfo=datetime.timezone.utc)
            return dt.timestamp()
        except (ValueError, ImportError):
            continue
    return None


def _fingerprint_match(raw, suffix, fingerprints):
    """Return the confidence float if a STRICT provider deleted-app fingerprint
    for this suffix matches the response body, else None.

    Strictness (integration F1, error-soundness F2/F5): only entries explicitly
    marked "strict": true fire CC (generic phrases like 'project not found' /
    'Application not found' were removed from the pack and any without the flag
    are ignored), and the token must match on non-alphanumeric boundaries — not
    as a loose substring buried inside a larger word — so a live app that merely
    happens to contain the phrase as part of its own content does not trip it."""
    entries = fingerprints.get(suffix)
    if not isinstance(entries, list):
        return None
    try:
        if isinstance(raw, (bytes, bytearray)):
            body = raw.decode("utf-8", errors="replace")
        else:
            body = str(raw or "")
    except Exception:
        return None
    for ent in entries:
        if not isinstance(ent, dict):
            continue
        if ent.get("strict") is not True:
            continue
        needle = ent.get("s")
        if not isinstance(needle, str) or not needle:
            continue
        pattern = r"(?<![0-9A-Za-z])" + re.escape(needle) + r"(?![0-9A-Za-z])"
        if re.search(pattern, body):
            conf = ent.get("confidence")
            return float(conf) if isinstance(conf, (int, float)) else 0.8
    return None
rea-analysis-optimizationmorluto/reaOptimize REA providers and shared analysis workflows for agent context, latency, CPU, and memory while preserving correctness and Evidence. Use for analysis bottlenecks and cache or query design; CI/test-suite optimization belongs to rea-ci-tests-audit.Stars: 472kUpdated Claude CodeCodexVerifiedInstall
rea-ci-tests-auditmorluto/reaAudit REA CI and test suites for correctness, fixture drift, false passes, and measured performance; use for suite or workflow investigations and efficiency changes, rather than ordinary feature testing or a single test failure.Stars: 472kUpdated Claude CodeCodexVerifiedInstall
rea-tool-designmorluto/reaAudit, investigate, design, or change REA investigation tools, CLI/MCP contracts, provider capabilities, and Evidence semantics.Stars: 472kUpdated Claude CodeCodexVerifiedInstall
ponytail-auditDietrichGebert/ponytailQuality audit of the whole repo: bugs, security, real load, missing tests, speed, and what to delete. Most important first.Stars: 196kUpdated Claude CodeCodexVerifiedInstall