Skip to content

alexgreensh/repo-forensics / skills/repo-forensics

repo-forensics

Flagged

Security forensics for git repos, AI skills, and MCP servers. Audits dependencies, detects prompt injection, credential theft, runtime dynamism, manifest drift, known CVEs, CISA KEV (actively exploited) vulns, and 2026 attack patterns. Not for fixing vulnerabilities or pentesting.

FileKindSize
.gitignoreother193 B
SKILL.mdskill md43.4 KB
checksums.jsonconfig10.8 KB
checksums.json.sigother64 B
data/README.mdmarkdown1.3 KB
data/compromised_versions.jsonconfig188.3 KB
data/rule_ids.csvother35.3 KB
data/rulepacks/.gitkeepother221 B
data/rulepacks/dead_anchors.jsonconfig9.0 KB
data/rulepacks/mcp_security.jsonconfig31.2 KB
data/rulepacks/runtime_dynamism.jsonconfig46.7 KB
data/rulepacks/sast.jsonconfig107.6 KB
data/rulepacks/secrets.jsonconfig30.4 KB
data/rulepacks/shared.jsonconfig4.4 KB
data/rulepacks/skill_threats.jsonconfig95.4 KB
data/runtime_floors.jsonconfig2.3 KB
data/yara/cryptominers.yarother1.8 KB
data/yara/hacktools.yarother2.2 KB
data/yara/malware.yarother2.2 KB
data/yara/manifest.jsonconfig8.0 KB
data/yara/webshells.yarother2.5 KB
references/cursor-hook-contract.mdmarkdown5.6 KB
references/kernel_privesc_sandbox_pack.mdmarkdown5.0 KB
references/mcp-attack-patterns.mdmarkdown8.7 KB
references/research_sources.mdmarkdown8.0 KB
references/threat_patterns.mdmarkdown20.3 KB
scripts/_context_gate.pyscript21.7 KB
scripts/_ed25519.pyscript5.1 KB
scripts/_pyc_unmarshal.pyscript17.1 KB
scripts/_shared_patterns.pyscript4.8 KB
scripts/adjudication.pyscript16.4 KB
scripts/adjudication_bridge.pyscript1.0 KB
scripts/aggregate_json.pyscript69.2 KB
scripts/auto_scan.pyexecutablescript152.1 KB
scripts/corpus_sync.pyscript10.0 KB
scripts/cursor_pre_scan.pyscript22.5 KB
scripts/dast_sandbox.sbother1.6 KB
scripts/dead_anchors_extract.pyscript14.8 KB
scripts/dead_anchors_probe.pyscript27.1 KB
scripts/forensics_core.pyscript176.5 KB
scripts/gen_rule_ids.pyscript8.6 KB
scripts/hook_adapter.pyscript19.4 KB
scripts/ioc_manager.pyscript49.6 KB
scripts/parse_pnpm_lock.pyscript7.9 KB
scripts/pre_scan.pyscript143.5 KB
scripts/pypi_pins.pyscript8.1 KB
scripts/refresh_controller.pyexecutablescript48.4 KB
scripts/refresh_threat_dbs.pyexecutablescript19.4 KB
scripts/rule_loader.pyscript44.8 KB
scripts/rulepack_feed.pyscript26.4 KB
scripts/run_forensics.shexecutablescript20.3 KB
scripts/sarif_export.pyscript18.6 KB
scripts/scan_agent_skills.pyscript33.2 KB
scripts/scan_archive.pyscript29.1 KB
scripts/scan_ast.pyscript44.9 KB
scripts/scan_binary.pyscript10.9 KB
scripts/scan_bytecode.pyscript28.2 KB
scripts/scan_dast.pyscript28.8 KB
scripts/scan_dataflow.pyscript15.4 KB
scripts/scan_dead_anchors.pyscript15.9 KB
scripts/scan_decode.pyscript37.5 KB
scripts/scan_dependencies.pyscript124.5 KB
scripts/scan_devcontainer.pyscript8.5 KB
scripts/scan_entropy.pyscript15.6 KB
scripts/scan_entrypoint.pyscript34.3 KB
scripts/scan_git_config.pyscript96.3 KB
scripts/scan_git_forensics.pyscript24.1 KB
scripts/scan_history.pyscript14.5 KB
scripts/scan_infra.pyscript54.3 KB
scripts/scan_integrity.pyscript41.0 KB
scripts/scan_lifecycle.pyscript65.1 KB
scripts/scan_manifest_drift.pyscript47.0 KB
scripts/scan_mcp_security.pyscript42.3 KB
scripts/scan_oversize.pyscript11.7 KB
scripts/scan_post_incident.pyscript36.9 KB
scripts/scan_provenance.pyscript19.4 KB
scripts/scan_runtime_dynamism.pyscript23.5 KB
scripts/scan_runtime_floors.pyscript33.5 KB
scripts/scan_sast.pyscript59.1 KB
scripts/scan_secrets.pyscript11.9 KB
scripts/scan_skill_threats.pyscript102.4 KB
scripts/scan_splitstream.pyscript29.5 KB
scripts/scan_yara.pyscript19.7 KB
scripts/session_scan.pyscript54.1 KB
scripts/suppressions.pyscript15.4 KB
scripts/validate_manifests.pyscript13.7 KB
scripts/verify_install.pyscript26.2 KB
scripts/vuln_feed.pyscript31.1 KB
tests/conftest.pyscript13.4 KB
tests/corpus/DEAD_ANCHORS_CORPUS.mdmarkdown2.3 KB
tests/corpus/budgets.jsonconfig5.6 KB
tests/cursor_helpers.pyscript12.6 KB
tests/fixtures/legacy_main_blocks.jsonconfig10.3 KB
tests/fixtures/powershell_verdict_corpus.jsonconfig114.9 KB
tests/golden/parity_mcp_security.jsonconfig17.5 KB
tests/golden/parity_runtime_dynamism.jsonconfig31.7 KB
tests/golden/parity_sast.jsonconfig59.8 KB
tests/golden/parity_secrets.jsonconfig18.8 KB
tests/golden/parity_skill_threats.jsonconfig48.8 KB
tests/parity_corpus.pyscript6.3 KB
tests/powershell_hook.pyscript8.2 KB
tests/report_builders.pyscript5.0 KB
tests/schemas/sarif-schema-2.1.0.jsonconfig109.1 KB
tests/shell_compat.pyscript2.2 KB
tests/test_adjudication_bridge.pyscript832 B
tests/test_adjudication_containment.pyscript1.4 KB
tests/test_adjudication_output.pyscript20.0 KB
tests/test_adjudication_suppressions.pyscript18.7 KB
tests/test_advisory_adjudication.pyscript1.3 KB
tests/test_agent_relay_rules.pyscript7.8 KB
tests/test_aggregate_json.pyscript10.0 KB
tests/test_auto_scan_patterns.pyscript10.5 KB
tests/test_benign_corpus.pyscript16.4 KB
tests/test_benign_new_scanners.pyscript6.1 KB
tests/test_build_output_skip.pyscript10.6 KB
tests/test_capability_ledger.pyscript18.8 KB
tests/test_codex_install.pyscript10.0 KB
tests/test_codex_marketplace.pyscript3.3 KB
tests/test_comment_false_positives.pyscript2.2 KB
tests/test_composer_dependencies.pyscript4.0 KB
tests/test_compromised_versions.pyscript51.2 KB
tests/test_context_gate.pyscript32.4 KB
tests/test_coverage_status.pyscript7.6 KB
tests/test_css_compounding_shrink.pyscript20.8 KB
tests/test_cursor_adapter.pyscript7.8 KB
tests/test_cursor_benign.pyscript3.4 KB
tests/test_cursor_degrade.pyscript12.7 KB
tests/test_cursor_install.pyscript17.1 KB
tests/test_cursor_latency.pyscript7.2 KB
tests/test_cursor_parity.pyscript5.8 KB
tests/test_cursor_redteam_fn.pyscript4.9 KB
tests/test_cursor_tamper_modify.pyscript14.2 KB
tests/test_cursor_wiring.pyscript32.3 KB
tests/test_d1_yara_correlation.pyscript9.2 KB
tests/test_dead_anchors_extract.pyscript7.6 KB
tests/test_dead_anchors_probe.pyscript19.7 KB
tests/test_evidence_engine_parity.pyscript8.1 KB
tests/test_evidence_model.pyscript11.4 KB
tests/test_forensics_core.pyscript79.2 KB
tests/test_fp_hardening_r3.pyscript9.7 KB
tests/test_fp_hardening_r4.pyscript18.8 KB
tests/test_fp_regression_pairs.pyscript62.5 KB
tests/test_fp_structural.pyscript21.2 KB
tests/test_freshness.pyscript6.2 KB
tests/test_frontmatter_validity.pyscript7.0 KB
tests/test_gpg_rce_hotfix.pyscript4.8 KB
tests/test_hook_inert_data.pyscript6.1 KB
tests/test_hook_scripts.pyscript14.0 KB
tests/test_inspection_cpu_budget.pyscript11.6 KB
tests/test_ioc_manager.pyscript55.4 KB
tests/test_kernel_privesc_rules.pyscript27.8 KB
tests/test_legit_precision_regressions.pyscript6.8 KB
tests/test_lethal_trifecta.pyscript20.7 KB
tests/test_manifest_signature.pyscript1.5 KB
tests/test_manifest_validity.pyscript2.9 KB
tests/test_monotonic_trust_e2e.pyscript4.4 KB
tests/test_non_breaking_contract.pyscript22.7 KB
tests/test_openclaw_install.pyscript6.6 KB
tests/test_pack_parity.pyscript24.8 KB
tests/test_parse_pnpm_lock.pyscript10.7 KB
tests/test_plugin_checkout_provenance.pyscript11.1 KB
tests/test_powershell_hook_wire.pyscript10.6 KB
tests/test_powershell_verdict_corpus.pyscript6.2 KB
tests/test_pr44_hardening.pyscript6.5 KB
tests/test_pre_scan.pyscript19.6 KB
tests/test_pre_scan_composer.pyscript7.4 KB
tests/test_precondition_fields.pyscript4.6 KB
tests/test_provenance_offline.pyscript2.5 KB
tests/test_published_rulepack_feed_current.pyscript1.4 KB
tests/test_py38_compat.pyscript3.3 KB
tests/test_pypi_pinned_versions.pyscript19.3 KB
tests/test_pyproject_string_forms.pyscript2.2 KB
tests/test_redteam_fn_regression.pyscript32.5 KB
tests/test_reflective_fetch_execute.pyscript23.5 KB
tests/test_reflective_review_fixes.pyscript17.8 KB
tests/test_refresh_controller.pyscript19.9 KB
tests/test_refresh_rulepacks.pyscript12.4 KB
tests/test_review_gate_regressions.pyscript9.3 KB
tests/test_rule_loader.pyscript23.2 KB
tests/test_rulepack_feed.pyscript38.1 KB
tests/test_run_forensics_json.pyscript1012 B
tests/test_run_scanner_fail_loud.pyscript6.5 KB
tests/test_runtime_floors.pyscript19.8 KB
tests/test_sandbox_probing_rules.pyscript9.6 KB
tests/test_sarif_export.pyscript19.9 KB
tests/test_scan_agent_skills.pyscript31.3 KB
tests/test_scan_archive.pyscript11.6 KB
tests/test_scan_ast.pyscript12.8 KB
tests/test_scan_binary.pyscript15.3 KB
tests/test_scan_bytecode.pyscript21.5 KB
tests/test_scan_dast.pyscript32.1 KB
tests/test_scan_dataflow.pyscript7.1 KB
tests/test_scan_dead_anchors.pyscript11.5 KB
tests/test_scan_decode.pyscript28.0 KB
tests/test_scan_dependencies.pyscript37.8 KB
tests/test_scan_devcontainer.pyscript8.5 KB
tests/test_scan_entropy.pyscript11.2 KB
tests/test_scan_entrypoint.pyscript19.5 KB
tests/test_scan_git_config.pyscript58.5 KB
tests/test_scan_git_config_review_fixes.pyscript33.3 KB

scripts/rule_loader.py

View on GitHub
#!/usr/bin/env python3
"""
rule_loader.py - Rules-as-data loader for repo-forensics v2.

Loads, validates, compiles, and self-tests JSON rule packs. A rule pack is a
declarative replacement for the regex/keyword/charset/map tables that used to
live hardcoded in scanner source. Scanners import this module and consume the
compiled structures; this module is a LEAF in the dependency graph (KTD-14):

    scanners  --->  rule_loader        (allowed)
    forensics_core / aggregate_json  --X-->  rule_loader   (FORBIDDEN)

Keeping `rule_loader` a leaf prevents circular imports — its compiled output
flows INTO Findings, it never reaches back into the aggregation layer.

Usage:
    from rule_loader import load_pack
    pack = load_pack("secrets")           # -> CompiledPack
    for rule in pack.all_rules:           # flat list (secrets/skill_threats/mcp)
        m = rule.regex.search(line)
    for rule in pack.by_extension.get(".py", ()):   # SAST hot loop, pre-indexed
        ...

Rule-pack JSON schema (mirrors data/compromised_versions.json conventions):
    {
      "schema_version": "1.0",          # "major.minor"; major-version gated
      "generated": "2026-06-10",
      "pack": "secrets",
      "pack_version": 1,                  # strictly-increasing INTEGER (feed overlay)
      "rules": [ {rule}, ... ]
    }

Each rule:
    {
      "id": "SC-KEY-001",                 # <SCANNER>-<CATEGORY>-<NNN>
      "type": "regex"|"keyword"|"charset"|"map",
      # --- type-specific payload ---
      "pattern": "...",                   # regex
      "flags": ["IGNORECASE", ...],       # regex (optional)
      "extensions": [".py", ".js"],       # regex (optional gate)
      "values": ["..."],                  # keyword (lowercase) / charset (codepoints)
      "mapping": {"а": "a"},              # map (e.g. homoglyph -> latin)
      # --- metadata (all types) ---
      "title": "...", "severity": "...", "confidence": 0.9,
      "category": "...", "explanation": "...",
      "examples": {"match": [...], "no_match": [...]},
      "retired": false                    # retired rules keep id, are skipped
    }

Security notes:
    * Packs resolve ONLY from the install-dir (realpath of this file) and the
      verified cache dir. Never from CWD, never relative, never from anything
      derived from a scan target. See _pack_search_paths().
    * ReDoS guards are MANDATORY (a compromised feed could ship a catastrophic
      pattern): pattern-length cap + nested-quantifier heuristic (both platform
      independent) + a hard per-rule self-test timeout (POSIX SIGALRM with
      save/restore, threading.Timer fallback elsewhere).

Created by Alex Greenshpun.
"""

import math
import os
import re
import sys
import json
import time
import signal
import threading
import unicodedata

# Keep sibling modules importable regardless of load path (CLI / hook / daemon).
_SCRIPTS_DIR = os.path.dirname(os.path.realpath(__file__))
if _SCRIPTS_DIR not in sys.path:
    sys.path.insert(0, _SCRIPTS_DIR)

# --- Schema / config --------------------------------------------------------

# Major-version gate, mirroring COMPROMISED_VERSIONS_SCHEMA_VERSION in
# ioc_manager.py. A pack whose major version differs is rejected wholesale;
# the caller treats that as "use the shipped fallback".
RULEPACK_SCHEMA_VERSION = "1.0"

# Valid rule types.
_RULE_TYPES = ("regex", "keyword", "charset", "map")

# Severity vocabulary (mirrors forensics_core); unknown severities are kept as
# given but warned — we do not silently rewrite a pack author's intent.
_KNOWN_SEVERITIES = ("critical", "high", "medium", "low", "info")

# Allowlisted re flag names. Anything outside this set is ignored with a warn,
# so a poisoned pack cannot smuggle in surprising behavior via flag strings.
_ALLOWED_REGEX_FLAGS = {
    "IGNORECASE": re.IGNORECASE,
    "MULTILINE": re.MULTILINE,
    "DOTALL": re.DOTALL,
    "VERBOSE": re.VERBOSE,
    "UNICODE": re.UNICODE,
    "ASCII": re.ASCII,
}

# --- ReDoS guards (mandatory, platform-independent) -------------------------

# Hard cap on regex source length. A legitimate detection pattern is well under
# this; a multi-kilobyte pattern is either machine-generated bloat or an attempt
# to defeat the nested-quantifier heuristic with sheer size.
MAX_PATTERN_LENGTH = 1000

# Per-rule self-test wall-clock budget (seconds). A rule whose examples cannot
# evaluate in this window is treated as catastrophic and rejected.
SELF_TEST_TIMEOUT_SEC = 2

# Nested-quantifier heuristic. Catches the classic catastrophic-backtracking
# shapes where a quantified group's BODY is itself a single quantified atom (or
# an alternation of such atoms) with no required literal boundary separating
# iterations — the exponential-backtracking signature:
#   (a+)+   (a*)*   (a+)*   ([ab]+)+   (\w+)+   (a|aa)+   (a+|b+)*
#
# It deliberately does NOT flag a quantified group whose body is a SEQUENCE
# containing a required, non-quantified separator, e.g. `(-[rf]+\s+)+` or
# `(-[uvzn\s]+)*` — there each iteration is anchored by structure, so it cannot
# blow up (verified empirically: those run sub-millisecond on adversarial input,
# while `(a+)+$` takes ~76s on 30 chars). Over-flagging real scanner patterns
# would block legitimate extraction in U3-U5, so the heuristic targets the
# precise dangerous shape rather than "any nested quantifier".
#
# Body grammar accepted as a "quantified atom": a char class `[...]`, an escape
# `\w`, or a literal char, optionally followed by a quantifier; branches joined
# by `|`. The whole body must be quantified atoms only (plus the outer quantifier).
_QUANT_ATOM = r"""(?: \\.|\[[^\]]*\]|[^()\[\]|*+?{}\\] ) [*+]? """
_NESTED_QUANT_RE = re.compile(
    r"""
    \(                          # group open
        (?: \?P?[<][^>]*> | \?: )?   # optional named/non-capturing prefix
        \s*
        """ + _QUANT_ATOM + r"""    # first atom (quantified)
        (?:                         # zero+ more atoms, all alternation branches
            \s* \| \s* """ + _QUANT_ATOM + r"""
        )*
    \)                          # group close
    \s*
    [*+]                        # ... and the GROUP itself is quantified
    """,
    re.VERBOSE,
)

# A quantified group whose body is a bare alternation (no inner quantifiers),
# e.g. `(a|aa)+`. Captures the alternation body so _looks_catastrophic can test
# whether any branch is a prefix of another (the overlap that makes it explode).
_ALT_GROUP_RE = re.compile(
    r"\((?:\?P?<[^>]*>|\?:)?\s*(?P<body>[^()]*\|[^()]*)\)\s*[*+]"
)

# Sequential-overlap quantifier heuristic (C1). Catches top-level runs of 3+
# consecutive quantified atoms of the same broad/overlapping class without a
# required literal boundary between them — the superpolynomial-backtracking
# signature when the suffix fails:
#
#   \w*\w*\w*\w*x   [a-z]*[a-z]*[a-z]*c   .*.*.*end
#   [^\n]*[^\n]*[^\n]*end   \w*\w+\w*x
#
# Three sub-cases, OR-ed together:
#   (a) Same backslash-class repeated 3+ times: \w*\w*\w*\w*
#       Uses a backreference so mixed classes like \w*\d*\s* (disjoint) don't fire.
#   (b) Same literal char-class body repeated 3+ times: [a-z]*[a-z]*[a-z]*
#       Same-text backreference catches overlapping positively/negated classes.
#   (c) Dot (.) repeated 3+ times: .*.*.*
#
# DELIBERATELY excluded:
#   - Bounded quantifiers {n,m}: [a-z]{0,5}[a-z]{0,5} are polynomial.
#   - Negated-class single appearances: [^)]*shell=True is linear.
#   - Mixed different classes: \w*\d*\s*end uses sub-(a)'s backreference to avoid.
#   - Lazy quantifiers are included (.*?.*?.*? is still catastrophic).
#
# Validation baseline: all 381 shipped-pack rules pass (zero false positives
# confirmed empirically).  The 5 known-evil patterns above are all caught.
_SEQ_QUANT_RE = re.compile(
    # (a) Same backslash-class escape repeated ×3+: \w*\w*\w*  or  \w*\w+\w*
    # Uses a backreference so mixed classes (\w*\d*\s*) don't fire.
    # NOTE: two backslashes in the raw string r"\\" = one literal backslash in
    # the Python string, which the regex engine then interprets as the escape
    # sequence prefix — matching a single \ followed by the class letter.
    # Four backslashes r"\\\\" would require TWO literal backslashes (\\w) in
    # the input, which never appears in real regex pattern strings.
    r"(\\[wWdDsS])[*+]\??\s*(?:\1[*+]\??\s*){2,}"
    r"|"
    r"(\[[^\]]{1,40}\])[*+]\??\s*(?:\2[*+]\??\s*){2,}"  # (b) same char-class ×3+
    r"|"
    r"\.[*+]\??\s*(?:\.[*+]\??\s*){2,}"  # (c) dot ×3+
)


def _looks_catastrophic(pattern):
    """Best-effort static detection of catastrophic-backtracking shapes.

    Returns a non-empty reason string if the pattern looks dangerous, else "".
    Platform-independent; complements (does not replace) the hard timeout, which
    is the real backstop for shapes a static heuristic can't see.

    Checks (in order):
      1. Length cap (MAX_PATTERN_LENGTH).
      2. Nested-quantifier shape: quantified group whose body is itself quantified,
         e.g. (a+)+, (\\w+)*.
      3. Overlapping-alternation in a quantified group, e.g. (a|aa)+.
      4. Sequential-overlap quantifiers at the top level: 3+ consecutive runs of
         the same broad class (\\w*\\w*\\w*x, [a-z]*[a-z]*[a-z]*c, .*.*.*end).
         This shape is missed by checks 2-3 but causes superpolynomial
         backtracking when the suffix fails (C1 fix).
    """
    if len(pattern) > MAX_PATTERN_LENGTH:
        return f"pattern length {len(pattern)} exceeds cap {MAX_PATTERN_LENGTH}"
    m = _NESTED_QUANT_RE.search(pattern)
    if m:
        # Only dangerous if an inner atom is itself quantified (overlap), i.e.
        # the matched body contains a `+`/`*` quantifier before the group
        # close. A bare `(abc)+` is fine; `(a+)+` is not.
        body = m.group(0)
        inner = body[:body.rfind(")")]
        if re.search(r"[*+]", inner):
            return "nested-quantifier shape (potential catastrophic backtracking)"
    # Overlapping-alternation inside a quantified group, e.g. `(a|aa)+` where one
    # branch is a prefix of another — an exponential shape with no inner
    # quantifier. Conservative: only flags simple-literal branches that share a
    # prefix, so `(?:foo|bar)+` (disjoint) is left alone. The hard timeout is the
    # backstop for anything subtler than this.
    for g in _ALT_GROUP_RE.finditer(pattern):
        branches = [b.strip() for b in g.group("body").split("|")]
        if len(branches) >= 2 and all(re.fullmatch(r"[^()\[\]|*+?{}\\]+", b or "x") for b in branches):
            for i in range(len(branches)):
                for j in range(len(branches)):
                    if i != j and branches[j] and branches[i].startswith(branches[j]):
                        return "overlapping-alternation in quantified group (potential catastrophic backtracking)"
    # Sequential-overlap quantifiers: 3+ consecutive runs of the same broad
    # quantified class without a required literal boundary (C1 fix).
    if _SEQ_QUANT_RE.search(pattern):
        return "sequential overlapping quantifiers (potential catastrophic backtracking)"
    return ""


# --- Timeout wrapper (platform-branched) ------------------------------------

_HAS_SIGALRM = hasattr(signal, "SIGALRM")


class RuleTestTimeout(Exception):
    """Raised when a rule's self-test exceeds SELF_TEST_TIMEOUT_SEC."""


def run_with_timeout(func, timeout=SELF_TEST_TIMEOUT_SEC):
    """Run `func()` under a hard wall-clock timeout, returning its result.

    Raises RuleTestTimeout if the budget is exceeded.

    POSIX (signal.SIGALRM available) — SAVE-AND-RESTORE style:
        We read the caller's pending alarm via `prev = signal.alarm(t)`. This
        is critical: refresh_threat_dbs.py arms its own 60s SIGALRM hard cap
        and THEN calls the feed updater, which (in U6) calls this loader's
        self-test. If we naively set/clear our alarm we would clobber that 60s
        cap. So in a `finally` we restore the caller's alarm to
        `max(1, prev - elapsed)` (or 0 if the caller had none). We also save and
        restore the previous SIGALRM handler.

    Windows / no SIGALRM — threading.Timer watchdog fallback:
        A daemon timer raises in the worker is not possible cleanly across
        threads, so we run `func` in the calling thread, start a Timer that
        sets a flag + interrupts via a cooperative check, and additionally cap
        by measuring elapsed time after the call. This is a WEAKER guarantee: a
        truly wedged C-level regexp match in CPython holds the GIL and the timer
        callback cannot preempt it. The static heuristic + length cap above are
        the real protection on this platform; the timer catches the common case
        and the post-hoc elapsed check rejects anything that overran.
    """
    if _HAS_SIGALRM:
        return _run_with_sigalrm(func, timeout)
    return _run_with_timer(func, timeout)


def _run_with_sigalrm(func, timeout):
    t = max(1, int(round(timeout)))

    def _handler(signum, frame):
        del signum, frame  # required by signal.signal contract; unused
        raise RuleTestTimeout(f"self-test exceeded {t}s (SIGALRM)")

    start = time.monotonic()
    prev_handler = signal.getsignal(signal.SIGALRM)
    # Arm OUR alarm; `prev` is the caller's remaining seconds (0 if none).
    prev = signal.alarm(0)  # read + clear caller's alarm without losing it
    signal.signal(signal.SIGALRM, _handler)
    signal.alarm(t)
    try:
        return func()
    finally:
        signal.alarm(0)
        signal.signal(signal.SIGALRM, prev_handler)
        # Restore the caller's hard cap, debited for the time we consumed.
        # If the debt has exhausted the caller's window, deliver the overdue
        # alarm on the soonest tick (1s) rather than alarm(0), which would
        # CANCEL it and silently drop the caller's timeout altogether.
        if prev > 0:
            elapsed = int(time.monotonic() - start)
            signal.alarm(max(1, prev - elapsed))
        # prev == 0 -> caller had no alarm; leave it disarmed.


def _run_with_timer(func, timeout):
    # No SIGALRM (Windows). We cannot preempt a wedged match holding the GIL, so
    # this is a weaker, advisory guard documented above. We run, then enforce a
    # post-hoc elapsed-time rejection; the Timer flag covers the cooperative case.
    timed_out = {"flag": False}

    def _trip():
        timed_out["flag"] = True

    timer = threading.Timer(timeout, _trip)
    timer.daemon = True
    start = time.monotonic()
    timer.start()
    try:
        result = func()
    finally:
        timer.cancel()
    elapsed = time.monotonic() - start
    if timed_out["flag"] or elapsed > timeout:
        raise RuleTestTimeout(
            f"self-test exceeded {timeout}s (timer fallback, elapsed {elapsed:.2f}s)"
        )
    return result


# --- Compiled structures ----------------------------------------------------

class CompiledRule:
    """One compiled, validated rule.

    Attributes (always present):
        id, type, title, severity, confidence, category, explanation
    Type-specific (exactly one set is meaningful per `type`):
        regex      -> .regex (compiled), .extensions (tuple or None)
        keyword    -> .values (tuple of NFKC-normalized lowercase strings)
        charset    -> .codepoints (frozenset[int])
        map        -> .mapping (dict[str, str])
    Plus .examples (dict with "match"/"no_match" lists) for self-testing.
    """

    __slots__ = (
        "id", "type", "title", "severity", "confidence", "category",
        "explanation", "examples", "regex", "extensions", "values",
        "codepoints", "mapping", "attacker", "boundary", "asset",
    )

    def __init__(self, *, id, type, title, severity, confidence, category,
                 explanation, examples, regex=None, extensions=None,
                 values=None, codepoints=None, mapping=None,
                 attacker="", boundary="", asset=""):
        self.id = id
        self.type = type
        self.title = title
        self.severity = severity
        self.confidence = confidence
        self.category = category
        self.explanation = explanation
        self.examples = examples or {"match": [], "no_match": []}
        self.regex = regex
        self.extensions = extensions  # tuple[str] or None (regex only)
        self.values = values          # tuple[str] (keyword)
        self.codepoints = codepoints  # frozenset[int] (charset)
        self.mapping = mapping        # dict[str, str] (map)
        self.attacker = attacker
        self.boundary = boundary
        self.asset = asset

    def __repr__(self):
        return f"<CompiledRule {self.id} type={self.type} sev={self.severity}>"


class CompiledPack:
    """A loaded, validated rule pack.

    Public surface consumed by scanners:
        .name           pack name (str)
        .pack_version   strictly-increasing integer
        .schema_version source schema string
        .source_path    install-dir or cache path it loaded from
        .all_rules      list[CompiledRule]  (flat — secrets/skill_threats/mcp)
        .by_extension   dict[str, list[CompiledRule]]  (regex rules, pre-indexed;
                        an empty-string key "" holds extension-agnostic regex
                        rules that apply to every file)
        .keyword_rules  list[CompiledRule]  (type == "keyword")
        .charset_rules  list[CompiledRule]  (type == "charset")
        .map_rules      list[CompiledRule]  (type == "map")
        .rules_for_extension(ext) -> list[CompiledRule]  (regex rules that apply
                        to `ext`: extension-agnostic rules + ext-gated ones)
    """

    def __init__(self, name, pack_version, schema_version, source_path, rules):
        self.name = name
        self.pack_version = pack_version
        self.schema_version = schema_version
        self.source_path = source_path
        self.all_rules = list(rules)

        self.by_extension = {}
        self.keyword_rules = []
        self.charset_rules = []
        self.map_rules = []

        for rule in self.all_rules:
            if rule.type == "regex":
                exts = rule.extensions if rule.extensions else ("",)
                for ext in exts:
                    self.by_extension.setdefault(ext, []).append(rule)
            elif rule.type == "keyword":
                self.keyword_rules.append(rule)
            elif rule.type == "charset":
                self.charset_rules.append(rule)
            elif rule.type == "map":
                self.map_rules.append(rule)

    def rules_for_extension(self, ext):
        """Regex rules applicable to files with extension `ext`.

        Returns extension-agnostic regex rules (key "") plus rules gated to
        `ext`. The SAST hot loop calls this once per file, so it stays
        O(rules-for-ext), never O(all-rules) per line.
        """
        out = list(self.by_extension.get("", ()))
        if ext and ext != "":
            out.extend(self.by_extension.get(ext, ()))
        return out

    def __repr__(self):
        return (f"<CompiledPack {self.name} v{self.pack_version} "
                f"rules={len(self.all_rules)}>")


# --- Validation / compilation -----------------------------------------------

def _warn(msg):
    print(f"[rule_loader] {msg}", file=sys.stderr)


def _normalize_keyword(s):
    """NFKC-normalize + lowercase a keyword value for substring matching."""
    return unicodedata.normalize("NFKC", s).lower()


def _parse_codepoint(v):
    """Accept an int codepoint or a 'U+XXXX'/'0xXXXX' string -> int, or None."""
    if isinstance(v, bool):
        return None  # bool is an int subclass; reject explicitly
    if isinstance(v, int):
        return v if 0 <= v <= 0x10FFFF else None
    if isinstance(v, str):
        s = v.strip()
        try:
            if s[:2].upper() == "U+":
                return int(s[2:], 16)
            if s[:2].lower() == "0x":
                return int(s, 16)
            # Single literal character is also acceptable.
            if len(s) == 1:
                return ord(s)
            return int(s)
        except (ValueError, IndexError):
            return None
    return None


def _compile_rule(raw):
    """Validate + compile one raw rule dict into a CompiledRule.

    Returns (CompiledRule, None) on success or (None, reason) on rejection.
    Structurally-invalid rules are skipped by the caller with a warning; they
    never crash the scanner (KTD-4 runtime-load granularity).
    """
    if not isinstance(raw, dict):
        return None, "rule is not an object"

    rule_id = raw.get("id")
    if not isinstance(rule_id, str) or not rule_id:
        return None, "missing/empty 'id'"

    if raw.get("retired") is True:
        return None, f"{rule_id}: retired"

    rtype = raw.get("type")
    if rtype not in _RULE_TYPES:
        return None, f"{rule_id}: invalid type {rtype!r}"

    severity = raw.get("severity", "medium")
    if not isinstance(severity, str):
        return None, f"{rule_id}: severity must be a string"
    if severity not in _KNOWN_SEVERITIES:
        _warn(f"{rule_id}: unknown severity {severity!r} (kept as-is)")

    confidence = raw.get("confidence", 0.0)
    try:
        confidence = float(confidence)
    except (TypeError, ValueError):
        # Non-numeric confidence (e.g. "high") — fall back to 0.0 so the caller
        # derives a severity-based default; do NOT reject the whole rule (C4).
        confidence = 0.0
    # NaN / ±inf compare False to every numeric test, so guard with isfinite (C4).
    if not math.isfinite(confidence) or confidence < 0.0:
        confidence = 0.0
    elif confidence > 1.0:
        confidence = 1.0

    title = raw.get("title", rule_id)
    category = raw.get("category", "")
    explanation = raw.get("explanation", "")
    examples = raw.get("examples", {}) or {}
    if not isinstance(examples, dict):
        examples = {}
    examples = {
        "match": [e for e in examples.get("match", []) if isinstance(e, str)],
        "no_match": [e for e in examples.get("no_match", []) if isinstance(e, str)],
    }

    common = dict(
        id=rule_id, type=rtype, title=str(title), severity=severity,
        confidence=confidence, category=str(category),
        explanation=str(explanation), examples=examples,
        attacker=str(raw.get("attacker", "")),
        boundary=str(raw.get("boundary", "")),
        asset=str(raw.get("asset", "")),
    )

    if rtype == "regex":
        pattern = raw.get("pattern")
        if not isinstance(pattern, str) or not pattern:
            return None, f"{rule_id}: regex rule missing 'pattern'"
        catastrophic = _looks_catastrophic(pattern)
        if catastrophic:
            return None, f"{rule_id}: rejected ({catastrophic})"
        flagval = 0
        for f in raw.get("flags", []) or []:
            if f in _ALLOWED_REGEX_FLAGS:
                flagval |= _ALLOWED_REGEX_FLAGS[f]
            else:
                _warn(f"{rule_id}: ignoring unknown regex flag {f!r}")
        try:
            compiled = re.compile(pattern, flagval)
        except re.error as e:
            return None, f"{rule_id}: re.error ({e})"
        exts = raw.get("extensions")
        if exts is not None:
            if not isinstance(exts, list):
                return None, f"{rule_id}: 'extensions' must be a list"
            exts = tuple(e for e in exts if isinstance(e, str) and e)
            if not exts:
                exts = None
        return CompiledRule(regex=compiled, extensions=exts, **common), None

    if rtype == "keyword":
        values = raw.get("values")
        if not isinstance(values, list) or not values:
            return None, f"{rule_id}: keyword rule missing non-empty 'values'"
        norm = tuple(_normalize_keyword(v) for v in values if isinstance(v, str) and v)
        if not norm:
            return None, f"{rule_id}: keyword rule has no usable values"
        return CompiledRule(values=norm, **common), None

    if rtype == "charset":
        values = raw.get("values")
        if not isinstance(values, list) or not values:
            return None, f"{rule_id}: charset rule missing non-empty 'values'"
        cps = set()
        for v in values:
            cp = _parse_codepoint(v)
            if cp is None:
                return None, f"{rule_id}: invalid codepoint {v!r}"
            cps.add(cp)
        return CompiledRule(codepoints=frozenset(cps), **common), None

    if rtype == "map":
        mapping = raw.get("mapping")
        if not isinstance(mapping, dict) or not mapping:
            return None, f"{rule_id}: map rule missing non-empty 'mapping'"
        clean = {k: v for k, v in mapping.items()
                 if isinstance(k, str) and isinstance(v, str)}
        if not clean:
            return None, f"{rule_id}: map rule has no usable mapping entries"
        return CompiledRule(mapping=clean, **common), None

    return None, f"{rule_id}: unhandled type {rtype!r}"  # pragma: no cover


# --- Self-test runner (reused by U6 feed verifier) --------------------------

class RuleTestResult:
    """Per-rule self-test outcome. Reusable by U6's feed verifier (don't assert
    inside the runner — return structured results so callers decide policy)."""

    __slots__ = ("rule_id", "passed", "failures")

    def __init__(self, rule_id, passed, failures):
        self.rule_id = rule_id
        self.passed = passed
        self.failures = failures  # list[str] of human-readable failure reasons

    def __repr__(self):
        status = "PASS" if self.passed else "FAIL"
        return f"<RuleTestResult {self.rule_id} {status} ({len(self.failures)} issues)>"


def _rule_matches(rule, text):
    """Does `rule` match `text`? Mirrors how scanners will consume each type."""
    if rule.type == "regex":
        return rule.regex.search(text) is not None
    if rule.type == "keyword":
        norm = _normalize_keyword(text)
        return any(v in norm for v in rule.values)
    if rule.type == "charset":
        return any(ord(ch) in rule.codepoints for ch in text)
    if rule.type == "map":
        return any(ch in rule.mapping for ch in text)
    return False  # pragma: no cover


def run_rule_self_test(rule, timeout=SELF_TEST_TIMEOUT_SEC):
    """Execute one rule's embedded match/no_match examples under a hard timeout.

    Returns a RuleTestResult. A rule whose `match` example does NOT match, or
    whose `no_match` example DOES match, fails (with the offending example named
    in .failures). A rule whose evaluation times out (catastrophic backtracking)
    fails with a timeout failure — the caller rejects it like a structurally
    invalid rule.

    This is the shared verifier: U6's whole-bundle feed acceptance reuses it.
    """
    failures = []
    try:
        def _do():
            local = []
            for ex in rule.examples.get("match", []):
                if not _rule_matches(rule, ex):
                    local.append(f"match example did not match: {ex!r}")
            for ex in rule.examples.get("no_match", []):
                if _rule_matches(rule, ex):
                    local.append(f"no_match example matched: {ex!r}")
            return local
        failures = run_with_timeout(_do, timeout=timeout)
    except RuleTestTimeout as e:
        failures = [f"self-test timeout: {e}"]
    return RuleTestResult(rule.id, not failures, failures)


def self_test_pack(pack, timeout=SELF_TEST_TIMEOUT_SEC):
    """Run every rule's self-test in a pack. Returns list[RuleTestResult].

    Pure data-out, no assertions — U6's feed verifier and the loader's
    own load-time validation both consume this.
    """
    return [run_rule_self_test(r, timeout=timeout) for r in pack.all_rules]


# --- Path discipline --------------------------------------------------------

# Install-dir rule packs live next to this script's parent: scripts/../data/rulepacks.
# Anchored via realpath(__file__) to resolve the ~/.claude/skills symlink — never
# from CWD, never relative, never from a scan target.
_INSTALL_RULEPACK_DIR = os.path.normpath(
    os.path.join(_SCRIPTS_DIR, "..", "data", "rulepacks")
)

# Cache overlay dir (U6 signed-feed overlay). The verified bundle written by
# rulepack_feed.accept_bundle lives here as bundle.json + bundle.json.sig.
# load_pack re-verifies the signature on EVERY load (KTD-12) and overlays a pack
# only when its cached pack_version is strictly newer than the shipped pack AND
# the schema major matches AND every example self-test passes.
CACHE_RULEPACK_DIR = os.path.join(
    os.path.expanduser("~"), ".cache", "repo-forensics", "rulepacks"
)
_CACHE_BUNDLE_FILE = "bundle.json"
_CACHE_BUNDLE_SIG_FILE = "bundle.json.sig"

# rule-pack-degraded flag (DISTINCT from ioc_manager's _ioc_degraded, KTD-11).
# Set True whenever a cache bundle exists but fails verification/overlay, so the
# scanner can surface a rule-pack-specific warning. Reset to False each time the
# overlay verifies cleanly or no cache is present at all.
_RULEPACK_DEGRADED = False
# Overlay log: human-readable notes about what the last overlay attempt did
# (which packs overlaid, any shipped-active rule arriving retired:true, any
# rejection reason). Surfaced by callers; never raises.
_OVERLAY_LOG = []


def get_rulepack_degraded():
    """True iff a rule-pack cache bundle is present but failed verification or
    overlay (tampered / invalid signature / rollback). Callers surface a
    rule-pack-degraded warning, distinct from the IOC-degraded message."""
    return _RULEPACK_DEGRADED


def get_overlay_log():
    """Return a copy of the most recent overlay log lines (diagnostics)."""
    return list(_OVERLAY_LOG)


def _set_degraded(value):
    global _RULEPACK_DEGRADED
    _RULEPACK_DEGRADED = bool(value)


def _overlay_note(msg):
    _OVERLAY_LOG.append(msg)


# Process-lifetime memo for the cache-bundle signature verification keyed on
# (path, mtime, size) so repeated loads in one process don't re-verify the same
# bytes needlessly (~5-20ms each, ~20 verifies per scan otherwise). KTD-12.
_BUNDLE_VERIFY_MEMO = {}


def _reset_overlay_state():
    """Test helper: clear overlay memo + degraded flag + log."""
    _BUNDLE_VERIFY_MEMO.clear()
    _OVERLAY_LOG.clear()
    _set_degraded(False)


def _verified_cache_bundle(cache_dir=None):
    """Read + verify the cached signed bundle. Returns the parsed bundle dict on
    success, or None (and sets the degraded flag) on any failure.

    Verify-on-load (KTD-12): the signature is checked over the EXACT cached raw
    bytes EVERY call — the cache is writable by any same-user process (the
    postinstall-malware class this tool detects). Memoized per (path, mtime,
    size); a tamper changes mtime/size and busts the memo, forcing a re-verify
    that then fails.
    """
    root = cache_dir if cache_dir is not None else CACHE_RULEPACK_DIR
    bundle_path = os.path.join(root, _CACHE_BUNDLE_FILE)
    sig_path = os.path.join(root, _CACHE_BUNDLE_SIG_FILE)
    previous_bundle_path = os.path.join(root, "bundle.previous.json")
    previous_sig_path = os.path.join(root, "bundle.previous.json.sig")
    if not ((os.path.isfile(bundle_path) and os.path.isfile(sig_path))
            or (os.path.isfile(previous_bundle_path) and os.path.isfile(previous_sig_path))):
        return None  # no complete generation at all -> shipped is the norm
    try:
        stats = []
        for path in (bundle_path, sig_path, previous_bundle_path, previous_sig_path):
            try:
                st = os.stat(path)
                stats.append((path, st.st_mtime_ns, st.st_size))
            except OSError:
                stats.append((path, None, None))
        memo_key = tuple(stats)
    except OSError:
        _set_degraded(True)
        return None
    if memo_key in _BUNDLE_VERIFY_MEMO:
        return _BUNDLE_VERIFY_MEMO[memo_key]
    # Import the verify chokepoint lazily (rulepack_feed -> _ed25519). KTD-14:
    # rulepack_feed/_ed25519 do NOT import rule_loader's aggregation consumers,
    # so this stays leaf-safe (it is the feed/crypto layer, not aggregation).
    try:
        import rulepack_feed
        current_pair = rulepack_feed._read_verified_pair(bundle_path, sig_path)
        pair = current_pair or rulepack_feed.read_verified_cached_pair(root)
    except Exception as e:  # never let a feed-module issue crash a scan
        _warn(f"cache bundle verify error: {e}")
        pair = None
        current_pair = None
    if pair is None:
        _warn("cached rule-pack bundle signature INVALID — ignoring cache, "
              "using shipped packs (rule-pack-degraded)")
        _set_degraded(True)
        _BUNDLE_VERIFY_MEMO[memo_key] = None
        return None
    raw, _sig = pair
    if current_pair is None:
        _warn("cached rule-pack current generation incomplete/invalid — using "
              "last-known-good signed generation")
        _set_degraded(True)
    try:
        bundle = json.loads(raw.decode("utf-8"))
        if not isinstance(bundle, dict):
            raise ValueError("bundle top-level not an object")
    except (ValueError, UnicodeDecodeError) as e:
        _warn(f"cached rule-pack bundle parse error after verify: {e}")
        _set_degraded(True)
        _BUNDLE_VERIFY_MEMO[memo_key] = None
        return None
    _BUNDLE_VERIFY_MEMO[memo_key] = bundle
    return bundle


def _compiled_pack_from_bundle_entry(name, entry, source_path):
    """Compile a pack from a verified-bundle entry into a CompiledPack, running
    every rule's self-test. Returns (CompiledPack, None) or (None, reason).
    Whole-pack acceptance: ANY self-test failure rejects the overlay for this
    pack (shipped stays authoritative)."""
    if not isinstance(entry, dict):
        return None, "bundle entry not an object"
    schema_version = entry.get("schema_version", RULEPACK_SCHEMA_VERSION)
    if not isinstance(schema_version, str):
        return None, "schema_version not a string"
    major = schema_version.split(".")[0] if schema_version else ""
    if major != RULEPACK_SCHEMA_VERSION.split(".")[0]:
        return None, f"schema major mismatch ({schema_version!r})"
    pack_version = entry.get("pack_version", 0)
    if isinstance(pack_version, bool) or not isinstance(pack_version, int):
        return None, "pack_version not an integer"
    raw_rules = entry.get("rules", [])
    if not isinstance(raw_rules, list):
        return None, "rules not a list"
    compiled = []
    seen = set()
    retired_ids = set()
    for raw in raw_rules:
        rule, reason = _compile_rule(raw)
        if rule is None:
            if reason and reason.endswith(": retired"):
                # Track retired ids so the overlay can flag silent removal of a
                # shipped-active rule (silent-detection-removal guard, KTD-6).
                rid = raw.get("id") if isinstance(raw, dict) else None
                if isinstance(rid, str) and rid:
                    retired_ids.add(rid)
                continue
            return None, f"rule rejected: {reason}"
        if rule.id in seen:
            continue
        seen.add(rule.id)
        result = run_rule_self_test(rule)
        if not result.passed:
            return None, (f"rule {rule.id} self-test failed: "
                          f"{'; '.join(result.failures)}")
        compiled.append(rule)
    cp = CompiledPack(name, pack_version, schema_version, source_path, compiled)
    cp._retired_ids = retired_ids  # attached for the silent-removal diff
    return cp, None


def _pack_search_paths(name, base_dir=None):
    """Resolve the candidate file path(s) for a pack `name`, in priority order.

    Security: every path is derived ONLY from realpath-anchored install dir (or
    an explicit `base_dir` passed by tests/U6), never from CWD or scan target.
    The pack name is sanitized to a bare filename so '../' traversal is
    impossible.

    U6 seam: when overlay support lands, the verified CACHE_RULEPACK_DIR is
    consulted here (newer pack_version + valid signature wins). For U2 we only
    return the install-dir path (or base_dir override).
    """
    safe = os.path.basename(str(name))
    if not safe or safe in (".", ".."):
        return []
    if not safe.endswith(".json"):
        safe = safe + ".json"
    root = base_dir if base_dir is not None else _INSTALL_RULEPACK_DIR
    return [os.path.join(root, safe)]


# --- Loading + memoization --------------------------------------------------

# Process-lifetime memo keyed on (resolved path, mtime). One process never
# parses the same pack file twice. Keyed on path (not just name) so a base_dir
# override or a future cache overlay is a distinct cache entry.
_PACK_CACHE = {}


def _reset_pack_cache():
    """Test helper: force a fresh parse on the next load_pack call."""
    _PACK_CACHE.clear()


class SchemaIncompatibleError(Exception):
    """Raised when a pack's schema major version is incompatible. Callers treat
    this as 'use the shipped fallback' (return None at the call site)."""


def _load_pack_file(path):
    """Parse + validate + compile a single pack file at `path`.

    Returns a CompiledPack. Raises:
        FileNotFoundError      - path missing
        SchemaIncompatibleError - major schema mismatch (whole-pack reject)
        ValueError              - structurally invalid top-level shape
    Individually-invalid rules are skipped with a stderr warning, never fatal.
    """
    with open(path, "r", encoding="utf-8") as f:
        data = json.load(f)

    if not isinstance(data, dict):
        raise ValueError(f"{path}: top-level must be a JSON object")

    # Schema major-version gate (mirror ioc_manager.py).
    schema_version = data.get("schema_version", "")
    if not isinstance(schema_version, str):
        raise SchemaIncompatibleError(
            f"{path}: schema_version must be a string, "
            f"got {type(schema_version).__name__}"
        )
    major = schema_version.split(".")[0] if schema_version else ""
    expected_major = RULEPACK_SCHEMA_VERSION.split(".")[0]
    if major != expected_major:
        raise SchemaIncompatibleError(
            f"{path}: schema version {schema_version!r} incompatible with "
            f"expected {RULEPACK_SCHEMA_VERSION!r}"
        )

    # pack_version is a strictly-increasing INTEGER (feed overlay comparison in
    # U6 uses explicit int comparison — no semver/string ambiguity per KTD-13).
    pack_version = data.get("pack_version", 0)
    if isinstance(pack_version, bool) or not isinstance(pack_version, int):
        raise ValueError(f"{path}: pack_version must be an integer")

    name = data.get("pack")
    if not isinstance(name, str) or not name:
        name = os.path.splitext(os.path.basename(path))[0]

    raw_rules = data.get("rules", [])
    if not isinstance(raw_rules, list):
        raise ValueError(f"{path}: 'rules' must be a list")

    compiled = []
    seen_ids = set()
    for raw in raw_rules:
        rule, reason = _compile_rule(raw)
        if rule is None:
            # Retired rules are an expected, silent skip; everything else warns.
            if not (reason and reason.endswith(": retired")):
                _warn(f"skipping rule in {os.path.basename(path)}: {reason}")
            continue
        if rule.id in seen_ids:
            _warn(f"duplicate rule id {rule.id} in {os.path.basename(path)}; "
                  f"keeping first")
            continue
        seen_ids.add(rule.id)
        # Runtime-load self-test granularity (KTD-4): a rule whose embedded
        # example fails (including a catastrophic-backtracking timeout) is
        # skipped individually, never crashing the scanner.
        result = run_rule_self_test(rule)
        if not result.passed:
            _warn(f"skipping rule {rule.id} ({os.path.basename(path)}): "
                  f"self-test failed: {'; '.join(result.failures)}")
            continue
        compiled.append(rule)

    return CompiledPack(name, pack_version, schema_version, path, compiled)


def _maybe_overlay(name, shipped, cache_dir=None):
    """Given the shipped CompiledPack for `name`, return the overlay pack from
    the verified cache bundle IF it is strictly newer and fully valid; otherwise
    return the shipped pack unchanged (shipped authoritative, KTD-6).

    Verifies the cache bundle's signature on every call (KTD-12, memoized).
    """
    bundle = _verified_cache_bundle(cache_dir)
    if bundle is None:
        return shipped  # no cache (or invalid -> degraded flag already set)
    packs = bundle.get("packs", {})
    if not isinstance(packs, dict):
        return shipped
    entry = packs.get(name)
    if not isinstance(entry, dict):
        return shipped  # bundle doesn't carry this pack
    entry_version = entry.get("pack_version", 0)
    shipped_version = shipped.pack_version if shipped is not None else -1
    if not isinstance(entry_version, int) or isinstance(entry_version, bool):
        return shipped
    if entry_version <= shipped_version:
        return shipped  # equal/older ignored — shipped stays authoritative
    overlay, reason = _compiled_pack_from_bundle_entry(
        name, entry, os.path.join(
            cache_dir if cache_dir is not None else CACHE_RULEPACK_DIR,
            _CACHE_BUNDLE_FILE)
    )
    if overlay is None:
        _warn(f"rule-pack overlay for {name!r} rejected: {reason} "
              f"(shipped v{shipped_version} stays authoritative)")
        _overlay_note(f"{name}: overlay REJECTED ({reason})")
        _set_degraded(True)
        return shipped
    # A newer feed may explicitly retire a shipped rule, but omission is not a
    # retirement decision. Keep the shipped pack when an overlay drops IDs.
    retired = getattr(overlay, "_retired_ids", set())
    if shipped is not None:
        shipped_ids = {r.id for r in shipped.all_rules}
        overlay_ids = {r.id for r in overlay.all_rules}
        omitted = sorted(shipped_ids - overlay_ids - retired)
        if omitted:
            detail = f"{len(omitted)} shipped-active rule(s) omitted: {', '.join(omitted[:5])}"
            _warn(f"rule-pack overlay for {name!r} rejected: {detail}")
            _overlay_note(f"{name}: overlay REJECTED ({detail})")
            _set_degraded(True)
            return shipped
    # Explicit retirement is surfaced under the existing KTD-6 policy.
    if shipped is not None and retired:
        silenced = sorted(shipped_ids & retired)
        for rid in silenced:
            _overlay_note(
                f"{name}: shipped-active rule {rid} retired by overlay "
                f"v{entry_version} (silent-detection-removal guard)")
            _warn(f"overlay retires shipped-active rule {rid} in pack {name!r}")
    _overlay_note(f"{name}: overlaid shipped v{shipped_version} -> v{entry_version}")
    return overlay


def load_pack(name, base_dir=None, cache_dir=None):
    """Load a rule pack by name. Returns a CompiledPack, or None if no pack file
    exists or the pack is schema-incompatible (caller falls back to shipped
    behavior).

    `base_dir` is a documented test/U6 seam: when supplied, packs resolve from
    that directory instead of the install dir. It is still resolved as an
    explicit absolute path — it is NEVER derived from a scan target or CWD.

    `cache_dir` overrides the signed-feed cache location (test seam). The
    verified cache bundle overlays a shipped pack only when strictly newer +
    valid (KTD-6); its signature is re-verified on every load (KTD-12).

    Memoized per (resolved path, mtime): one process never parses a pack twice.
    """
    for path in _pack_search_paths(name, base_dir=base_dir):
        if not os.path.isfile(path):
            continue
        try:
            mtime = os.path.getmtime(path)
        except OSError:
            continue
        cache_key = (path, mtime)
        if cache_key in _PACK_CACHE:
            shipped = _PACK_CACHE[cache_key]
        else:
            try:
                shipped = _load_pack_file(path)
            except SchemaIncompatibleError as e:
                _warn(f"{e} — falling back to shipped behavior")
                return None
            except (OSError, ValueError, json.JSONDecodeError) as e:
                _warn(f"could not load pack {name!r}: {e}")
                return None
            _PACK_CACHE[cache_key] = shipped
        # Overlay seam: a base_dir override (tests loading a single pack file)
        # bypasses the cache overlay entirely — the override IS the source of
        # truth. The signed-feed overlay only applies to install-dir loads.
        if base_dir is not None:
            return shipped
        return _maybe_overlay(name, shipped, cache_dir=cache_dir)
    return None


# --- CLI (dev convenience: self-test a pack) --------------------------------

def main(argv=None):
    argv = argv if argv is not None else sys.argv[1:]
    if not argv:
        print("usage: rule_loader.py <pack-name|path-to-pack.json>", file=sys.stderr)
        return 2
    target = argv[0]
    if os.path.isfile(target):
        base_dir = os.path.dirname(os.path.abspath(target))
        name = os.path.basename(target)
    else:
        base_dir = None
        name = target
    pack = load_pack(name, base_dir=base_dir)
    if pack is None:
        print(f"[rule_loader] pack {target!r} not found or incompatible",
              file=sys.stderr)
        return 1
    results = self_test_pack(pack)
    failed = [r for r in results if not r.passed]
    print(f"pack={pack.name} v{pack.pack_version} rules={len(pack.all_rules)} "
          f"self-test: {len(results) - len(failed)} pass, {len(failed)} fail")
    for r in failed:
        print(f"  FAIL {r.rule_id}: {'; '.join(r.failures)}", file=sys.stderr)
    return 0 if not failed else 1


if __name__ == "__main__":
    sys.exit(main())
rea-analysis-optimizationmorluto/reaOptimize REA providers and shared analysis workflows for agent context, latency, CPU, and memory while preserving correctness and Evidence. Use for analysis bottlenecks and cache or query design; CI/test-suite optimization belongs to rea-ci-tests-audit.Stars: 472kUpdated Claude CodeCodexVerifiedInstall
rea-ci-tests-auditmorluto/reaAudit REA CI and test suites for correctness, fixture drift, false passes, and measured performance; use for suite or workflow investigations and efficiency changes, rather than ordinary feature testing or a single test failure.Stars: 472kUpdated Claude CodeCodexVerifiedInstall
rea-tool-designmorluto/reaAudit, investigate, design, or change REA investigation tools, CLI/MCP contracts, provider capabilities, and Evidence semantics.Stars: 472kUpdated Claude CodeCodexVerifiedInstall
ponytail-auditDietrichGebert/ponytailQuality audit of the whole repo: bugs, security, real load, missing tests, speed, and what to delete. Most important first.Stars: 194kUpdated Claude CodeCodexVerifiedInstall