Skip to content

hypnguyen1209/offensive-claude / skills/vulnerability-analysis

vulnerability-analysis

Unverified

Use when auditing source code for vulnerabilities — drive CodeQL/Semgrep/Joern to taint untrusted data source-to-sink across injection, memory safety, deserialization/prototype-pollution, secrets/crypto/authz/race, and supply-chain risks

scripts/validate_findings.py

View on GitHub
#!/usr/bin/env python3
"""validate_findings.py - evidence-grounding + false-positive harness for findings.

Enforces the CLAUDE.md mandate "assess real exploitability (not just pattern matching)"
mechanically, for LIVE/dynamic findings (the static-audit counterpart is
skills/references/taint-false-positive-reduction.md).

DESIGN: tiers are driven by STRUCTURED proof signals (explicit booleans), never by regex
over analyst prose. Prose paraphrase can flip a substring match in either direction
(a self-IDOR worded to dodge a kill phrase, or a real RCE rejected because it says "the
initial probe returned no output"). Booleans can't be paraphrased. The job of judging
whether a boolean is HONEST belongs to the adversarial finding-validator agent /
human review - this tool is the mechanical gate.

Two checks per finding:
  1. GROUNDING   - every cited evidence path must exist on disk and be non-empty.
  2. CLASS PROOF - for a recognized vuln class, CONFIRMED requires the class's positive
                   proof signal to be explicitly true; an explicit disqualifier signal
                   forces REJECTED; otherwise the finding is POSSIBLE (gather more proof).

Output tiers:  CONFIRMED | POSSIBLE | INFO | REJECTED   (see finding-validation-runtime.md)

Finding input = JSON list. `proof` is an OBJECT of boolean signals:
  [{"id":"F1","title":"SSRF in webhook fetch","cwe":"CWE-918","severity":"High",
    "evidence":["logs/F1.txt"],
    "proof":{"internal_response_read": true}}]

Per-class signals:
  SSRF (918)              confirm: internal_response_read
  IDOR/BOLA (639/862/...) confirm: cross_identity_confirmed   disqualify: self_idor
  CORS (942)              confirm: creds_reflected_origin
  Open redirect (601)     confirm: external_redirect          disqualify: same_origin
  XSS (79/80)             confirm: script_executed            disqualify: encoded_inert
  RCE/cmd/deser (78/77/94/502) confirm: command_output_captured

Optional EVD-citation grounding (raptor-adoption): with --evidence-store evidence.json (an
evidence_kit store), any [EVD-XXX] a finding cites must EXIST and be VERIFIED, else the finding is
REJECTED (fail-closed - the mechanical half of finding-validator Q2/Q2.5). Hedge phrases
("probably", "should work", "likely exploitable") are linted as advisory warnings; --strict makes a
hedge/uncited warning on a CONFIRMED finding also fail the gate. Default behavior is unchanged.

CLI:
  validate_findings.py --findings findings.json [--evidence ./evidence]
                       [--evidence-store evidence.json] [--strict] [--json]
  exit 0 = clean; exit 1 = at least one REJECTED (or --strict lint fail); exit 2 = error
"""
from __future__ import annotations

import argparse
import json
import re
import sys
from dataclasses import dataclass, asdict, field
from pathlib import Path
from typing import Optional

sys.path.insert(0, str(Path(__file__).resolve().parent))
import provenance  # noqa: E402  (sibling module: per-hop reachability provenance gate)

# Hedge phrases: ungrounded-language smell. ADVISORY only - they never drive a tier (prose is
# paraphrasable; only structured signals confirm). They surface "you wrote it like a guess" so a
# reviewer re-checks, and in --strict a CONFIRMED finding carrying them fails the lint gate.
HEDGE_PHRASES = [
    r"\bprobably\b", r"\bshould\s+(?:be|work)\b", r"\blikely\s+exploitabl", r"\bmight\b",
    r"\bmaybe\b", r"\bseems?\s+to\b", r"\bappears?\s+to\b", r"\bin\s+theory\b",
    r"\bcould\s+(?:potentially|possibly)\b", r"\bi\s+(?:think|believe|assume)\b",
    r"\btheoretically\b", r"\bpresumabl", r"\bguess", r"\bnot\s+sure\b",
]
_HEDGE_RX = [re.compile(p, re.I) for p in HEDGE_PHRASES]
_EVD_RX = re.compile(r"EVD-[A-Za-z0-9_.\-]+", re.I)   # case-insensitive: 'evd-001' can't dodge the gate

# class -> cwe ids, title keywords (fallback class detection), confirm signal, disqualify signal
CLASS_RULES = [
    {"name": "SSRF", "cwe": {"918"}, "kw": ["ssrf", "server-side request"],
     "confirm": "internal_response_read", "disqualify": None,
     "hint": "an internal/restricted response (not a bare DNS/OOB callback) was read back"},
    {"name": "IDOR/BOLA", "cwe": {"639", "862", "566", "284", "1220"}, "kw": ["idor", "bola", "broken object", "broken access"],
     "confirm": "cross_identity_confirmed", "disqualify": "self_idor",
     "hint": "another principal's data was accessed using a second identity you control"},
    {"name": "CORS", "cwe": {"942"}, "kw": ["cors", "cross-origin resource"],
     "confirm": "creds_reflected_origin", "disqualify": None,
     "hint": "ACAO reflected the origin AND Access-Control-Allow-Credentials:true, with a cross-origin authenticated read"},
    {"name": "Open Redirect", "cwe": {"601"}, "kw": ["open redirect", "open-redirect"],
     "confirm": "external_redirect", "disqualify": "same_origin",
     "hint": "the final redirect Location host is external/attacker-controlled"},
    {"name": "XSS", "cwe": {"79", "80"}, "kw": ["xss", "cross-site scripting"],
     "confirm": "script_executed", "disqualify": "encoded_inert",
     "hint": "script actually executed in the victim context (not reflected-but-encoded)"},
    {"name": "RCE/Command Injection", "cwe": {"78", "77", "94", "502"}, "kw": ["rce", "command inj", "remote code", "deserial"],
     "confirm": "command_output_captured", "disqualify": None,
     "hint": "command output was captured (blind with no confirming oob is POSSIBLE, not CONFIRMED)"},
]

# Native memory-corruption classes need a machine-checkable REACHABILITY artifact to CONFIRM - a
# crash on paper is not demonstrated reachability. Without a gcov line-hit or a function trace the
# finding is POSSIBLE (the vuln line may never execute on the path you think). raptor-adoption F/F.
NATIVE_MEMORY_CWES = {
    "119", "120", "121", "122", "124", "125", "126", "127", "129", "131", "134", "170",
    "190", "191", "415", "416", "457", "476", "562", "590", "680", "763",
    "786", "787", "788", "805", "822", "823", "824", "825", "843",
}


@dataclass
class Verdict:
    id: str
    title: str
    tier: str           # CONFIRMED | POSSIBLE | INFO | REJECTED
    reason: str
    cwe: str = ""
    matched_class: Optional[str] = None
    lint: list = field(default_factory=list)   # advisory warnings (hedges, uncited) - not a tier

    def to_dict(self) -> dict:
        return asdict(self)


def _iter_strings(obj):
    """Yield every string reachable in a finding record - dict KEYS and values, list items - so a
    citation/hedge can't hide in a key position (or any nested field) to dodge the scan."""
    if isinstance(obj, str):
        yield obj
    elif isinstance(obj, dict):
        for k, v in obj.items():
            if isinstance(k, str):
                yield k
            yield from _iter_strings(v)
    elif isinstance(obj, (list, tuple)):
        for v in obj:
            yield from _iter_strings(v)


def _cited_evds(f: dict) -> set:
    cited = set()
    for s in _iter_strings(f):
        cited.update(m.upper() for m in _EVD_RX.findall(s))   # normalize case for comparison
    return cited


def _lint_hedges(f: dict) -> list:
    """Hedge phrases anywhere in the finding's strings. Advisory: a confidence smell, never a
    verdict. Scans every field (not just title/description) so a hedge in poc/recommendation/notes
    still surfaces under --strict."""
    hits = set()
    for text in _iter_strings(f):
        for rx in _HEDGE_RX:
            m = rx.search(text)
            if m:
                hits.add(m.group(0).lower().strip())
    return sorted(hits)


def _cwe_num(cwe) -> str:
    m = re.search(r"(\d+)", str(cwe or ""))
    return m.group(1) if m else ""


def _grounded(evidence, evidence_dir: Optional[Path]) -> tuple[bool, str]:
    paths = evidence if isinstance(evidence, list) else ([evidence] if evidence else [])
    if not paths:
        return False, "finding cites no evidence artifact"
    for p in paths:
        if not isinstance(p, str):
            return False, f"evidence entry is not a path string: {p!r}"
        candidates = [Path(p)]
        if evidence_dir:
            candidates.append(evidence_dir / p)
        ok = False
        for c in candidates:
            try:
                if not (c.is_file() and c.stat().st_size > 0):
                    continue
                if evidence_dir is not None and not _within(c, evidence_dir):
                    continue                   # outside the evidence dir -> not grounded
                ok = True
                break
            except OSError:
                continue
        if not ok:
            return False, f"evidence not found, empty, or outside the evidence dir: {p}"
    return True, "all cited evidence present"


def _match_rule(cwe_num: str, title: str):
    if cwe_num:
        for rule in CLASS_RULES:
            if cwe_num in rule["cwe"]:
                return rule
    t = (title or "").lower()
    for rule in CLASS_RULES:
        if any(k in t for k in rule["kw"]):
            return rule
    return None


def _within(child: Path, root: Path) -> bool:
    try:
        rc, rr = child.resolve(), root.resolve()
    except OSError:
        return False
    return rc == rr or rr in rc.parents


def _resolve(p, evidence_dir: Optional[Path]):
    """Resolve an evidence/proof path. When evidence_dir is set, a candidate is accepted ONLY if it
    lives inside it - an absolute path or '..' escape outside the evidence dir is rejected, so every
    proof artifact is tied to the engagement's evidence set."""
    if not isinstance(p, str) or not p:
        return None
    cands = [Path(p)] + ([evidence_dir / p] if evidence_dir else [])
    for c in cands:
        try:
            if not (c.is_file() and c.stat().st_size > 0):
                continue
            if evidence_dir is not None and not _within(c, evidence_dir):
                continue                       # outside the evidence dir -> not grounded
            return c
        except OSError:
            continue
    return None


def _check_gcov(cov: dict, evidence_dir: Optional[Path]) -> tuple:
    """A gcov line-hit proof: the vulnerable source line is NOT '#####' (never executed)."""
    path, line = cov.get("gcov"), cov.get("line")
    if not isinstance(line, int):
        return False, "coverage_proof.line must be an integer"
    fpath = _resolve(path, evidence_dir)
    if fpath is None:
        return False, f"coverage_proof gcov not found/empty: {path!r}"
    try:
        text = fpath.read_text(encoding="utf-8", errors="replace")
    except OSError:
        return False, f"coverage_proof gcov unreadable: {path!r}"
    for raw in text.splitlines():
        parts = raw.split(":", 2)               # gcov format: "<count>:<lineno>:<source>"
        if len(parts) < 2:
            continue
        count_s, lineno_s = parts[0].strip(), parts[1].strip()
        if lineno_s == str(line):
            # ALLOWLIST: a real gcov execution count is a positive integer. Reject everything else -
            # '#####'/'-'/'=====' (never executed), '0' (zero executions), and any forged/garbage
            # token ('exec', '-5', 'banana') that a blocklist would have let through (fail-open).
            shown = count_s
            count_s = count_s.replace(",", "").rstrip("*")   # thousands sep; '*' = executed, has a 0-count block
            # ASCII decimal ONLY (re.fullmatch r"[0-9]+", not str.isdigit which accepts fullwidth/
            # Arabic-Indic Unicode digits) and strictly positive.
            if re.fullmatch(r"[0-9]+", count_s) and int(count_s) > 0:
                return True, f"gcov shows line {line} executed (count={count_s})"
            return False, f"gcov line {line} not executed (count={shown!r}; need a positive integer hit)"
    return False, f"line {line} not present in gcov {path!r}"


def _check_trace(tr: dict, evidence_dir: Optional[Path]) -> tuple:
    """A function-trace proof: the vulnerable function name appears in a runtime trace log."""
    path, fn = tr.get("log"), tr.get("function")
    if not isinstance(fn, str) or not fn:
        return False, "trace_proof.function missing"
    fpath = _resolve(path, evidence_dir)
    if fpath is None:
        return False, f"trace_proof log not found/empty: {path!r}"
    try:
        text = fpath.read_text(encoding="utf-8", errors="replace")
    except OSError:
        return False, f"trace_proof log unreadable: {path!r}"
    # A trace proof must evidence EXECUTION, not a textual mention. We require a STRUCTURAL call
    # record - the function with an argument paren `fn(` (ltrace/uftrace/frame), a gdb/rr backtrace
    # frame `#N ... fn`, or a trace arrow `-> fn` / `>>> fn`. We do NOT match English keywords like
    # "call"/"enter" as substrings: those appear in prose ("never called", "recall", "ENTERPRISE",
    # "do not invoke fn") and would fail open. Comment/prose lines (#, //, *, ;) are skipped.
    fnre = re.escape(fn)
    word = re.compile(r"\b" + fnre + r"\b")
    call = re.compile(r"\b" + fnre + r"\s*\(")               # fn( ... )  -> a call / frame
    arrow = re.compile(r"(?:>>>|->|=>)\s*" + fnre + r"\b")    # uftrace / trace arrow into fn
    frame = re.compile(r"^#\d+\b.*\b" + fnre + r"\b")         # gdb/rr backtrace frame (authoritative)
    for raw in text.splitlines():
        line = raw.strip()
        if not word.search(line):
            continue
        if frame.match(line):
            return True, f"trace shows {fn!r} in a stack frame"
        if line[:2] in ("//", "/*") or line[:1] in ("#", ";", "*"):
            continue                                          # comment/prose line - not an exec record
        if call.search(line) or arrow.search(line):
            return True, f"trace shows {fn!r} in an execution record (call/frame/arrow)"
    return False, (f"function {fn!r} not found in an EXECUTION context in {path!r} - needs a call "
                   "'fn(...)', a stack frame '#N ... fn', or a trace arrow; a bare mention in "
                   "prose/comment/symbol-dump is not reachability")


def _native_reachability(signals: dict, evidence_dir: Optional[Path]) -> tuple:
    """Native memory-corruption needs a reachability artifact: a gcov line-hit OR a function trace.
    Returns (ok, why). Fail-closed: a malformed/missing artifact is NOT reachability."""
    cov = signals.get("coverage_proof")
    cov_reason = "no coverage_proof"
    if isinstance(cov, dict):
        ok, why = _check_gcov(cov, evidence_dir)
        if ok:
            return True, why
        cov_reason = why
    tr = signals.get("trace_proof")
    tr_reason = "no trace_proof"
    if isinstance(tr, dict):
        ok, why = _check_trace(tr, evidence_dir)
        if ok:
            return True, why
        tr_reason = why
    return False, (f"no demonstrated reachability ({cov_reason}; {tr_reason}) - provide "
                   "proof.coverage_proof={gcov,line} with a non-'#####' hit, or "
                   "proof.trace_proof={log,function} naming the executed vulnerable function")


def _apply_provenance_cap(verdict: Verdict, f: dict) -> Verdict:
    """PR-3/T2 per-hop provenance gate. If a finding declares a `reachability` path, a CONFIRMED
    verdict is capped to POSSIBLE unless EVERY hop is EXTRACTED (graphify weakest-hop rule). This is
    fail-closed (unknown/missing per-hop provenance => AMBIGUOUS) and there is NO promotion path -
    a reachability claim can only *lower* the tier, never raise it."""
    reach = f.get("reachability")
    if reach is None:
        return verdict                      # no path declared -> unchanged (live findings, etc.)
    if verdict.tier != "CONFIRMED":
        return verdict                      # cap only ever downgrades; never touches POSSIBLE/INFO/REJECTED
    if provenance.all_extracted(reach):
        return verdict                      # all-EXTRACTED path -> CONFIRMED stands
    weakest = provenance.weakest(reach)
    verdict.tier = "POSSIBLE"
    verdict.reason = (f"reachability weakest hop is {weakest} (not all-EXTRACTED) -> capped to "
                      f"POSSIBLE; {verdict.reason}")
    return verdict


def evaluate_finding(f: dict, evidence_dir: Optional[Path], *,
                     evidence_ids: Optional[set] = None,
                     verified_ids: Optional[set] = None) -> Verdict:
    """Public entry: run the mechanical class/grounding gate, then apply the provenance cap."""
    verdict = _evaluate_core(f, evidence_dir, evidence_ids=evidence_ids, verified_ids=verified_ids)
    if isinstance(f, dict):
        verdict = _apply_provenance_cap(verdict, f)
    return verdict


def _evaluate_core(f: dict, evidence_dir: Optional[Path], *,
                   evidence_ids: Optional[set] = None,
                   verified_ids: Optional[set] = None) -> Verdict:
    if not isinstance(f, dict):           # a malformed (non-object) element must not crash the run
        return Verdict("?", "", "REJECTED", f"malformed finding: not a JSON object ({type(f).__name__})", "")
    fid = str(f.get("id", "?"))
    title = str(f.get("title", ""))
    cwe = str(f.get("cwe", ""))
    cwe_num = _cwe_num(cwe)
    severity = str(f.get("severity", "")).lower()
    proof = f.get("proof")
    signals = proof if isinstance(proof, dict) else {}
    lint = [f"hedge: '{h}'" for h in _lint_hedges(f)]

    ok, why = _grounded(f.get("evidence"), evidence_dir)
    if not ok:
        return Verdict(fid, title, "REJECTED", f"ungrounded: {why}", cwe, lint=lint)

    # EVD-citation grounding (only when an evidence store was supplied). A cited evidence id that
    # is unknown (dangling) or not VERIFIED breaks grounding -> REJECTED, fail-closed. This is the
    # mechanical half of finding-validator Q2/Q2.5 (grounded + authentic at source).
    if evidence_ids is not None:
        cited = _cited_evds(f)
        dangling = sorted(cited - evidence_ids)
        if dangling:
            return Verdict(fid, title, "REJECTED",
                           f"cites unknown evidence id(s): {', '.join(dangling)}", cwe, lint=lint)
        if verified_ids is not None:
            unverified = sorted(cited - verified_ids)
            if unverified:
                return Verdict(fid, title, "REJECTED",
                               f"cites non-VERIFIED evidence id(s): {', '.join(unverified)} "
                               "(run evidence_kit.py verify)", cwe, lint=lint)
        if not cited:
            lint.append("uncited: no [EVD-XXX] citation despite an evidence store")

    rule = _match_rule(cwe_num, title)
    if rule:
        if rule["disqualify"] and signals.get(rule["disqualify"]) is True:
            return Verdict(fid, title, "REJECTED",
                           f"{rule['name']} disqualified: proof.{rule['disqualify']} is true", cwe, rule["name"], lint)
        if signals.get(rule["confirm"]) is True:
            if severity in ("info", "informational"):
                return Verdict(fid, title, "INFO", f"{rule['name']} proven but informational", cwe, rule["name"], lint)
            return Verdict(fid, title, "CONFIRMED",
                           f"{rule['name']} impact proven via proof.{rule['confirm']} and grounded", cwe, rule["name"], lint)
        kind = "set" if isinstance(proof, dict) else "provide a structured proof object and set"
        return Verdict(fid, title, "POSSIBLE",
                       f"{rule['name']}: not confirmed - {kind} proof.{rule['confirm']}=true once {rule['hint']}",
                       cwe, rule["name"], lint)

    # Native memory-corruption: a crash on paper is not exploitability - require a reachability artifact.
    if cwe_num in NATIVE_MEMORY_CWES:
        ok, why = _native_reachability(signals, evidence_dir)
        if not ok:
            return Verdict(fid, title, "POSSIBLE", f"native memory-corruption: {why}",
                           cwe, "Native Memory-Corruption", lint)
        if severity in ("info", "informational"):
            return Verdict(fid, title, "INFO", f"native bug reachable ({why}) but informational",
                           cwe, "Native Memory-Corruption", lint)
        return Verdict(fid, title, "CONFIRMED",
                       f"native memory-corruption with demonstrated reachability: {why}",
                       cwe, "Native Memory-Corruption", lint)

    if severity in ("info", "informational"):
        return Verdict(fid, title, "INFO", "grounded; informational severity", cwe, lint=lint)
    return Verdict(fid, title, "CONFIRMED", "grounded; no automated class check applies (reviewer must confirm impact)", cwe, lint=lint)


def load_findings(path: str) -> list[dict]:
    with open(path, "r", encoding="utf-8") as fh:
        data = json.load(fh)
    if isinstance(data, dict):
        data = data.get("findings", [])
    if not isinstance(data, list):
        raise ValueError("findings file must be a JSON list (or {findings:[...]})")
    return data


def main(argv: Optional[list[str]] = None) -> int:
    p = argparse.ArgumentParser(description="Evidence-grounding + false-positive harness for findings.")
    p.add_argument("--findings", required=True, help="JSON list of finding records")
    p.add_argument("--evidence", help="directory holding evidence artifacts (paths resolved against it)")
    p.add_argument("--evidence-store", help="evidence_kit JSON; enables [EVD-XXX] citation grounding")
    p.add_argument("--strict", action="store_true",
                   help="lint warnings (hedges / uncited) on a CONFIRMED finding also fail the gate")
    p.add_argument("--json", action="store_true")
    args = p.parse_args(argv)

    try:
        findings = load_findings(args.findings)
    except (OSError, ValueError, json.JSONDecodeError) as exc:
        print(f"error: {exc}", file=sys.stderr)
        return 2

    evidence_ids = verified_ids = None
    if args.evidence_store:
        try:
            import evidence_kit
            store = evidence_kit.EvidenceStore.load(args.evidence_store)
            store.verify_all()                       # re-verify integrity before trusting citations
            evidence_ids = {i.upper() for i in store.ids()}        # compare case-insensitively
            verified_ids = {i.upper() for i in store.verified_ids()}
        except Exception as exc:                     # a broken store must not silently disable the check
            print(f"error: --evidence-store: {exc}", file=sys.stderr)
            return 2

    ev_dir = Path(args.evidence) if args.evidence else None
    verdicts = [evaluate_finding(f, ev_dir, evidence_ids=evidence_ids, verified_ids=verified_ids)
                for f in findings]

    if args.json:
        print(json.dumps([v.to_dict() for v in verdicts], indent=2))
    else:
        for v in verdicts:
            tag = f"  [lint: {'; '.join(v.lint)}]" if v.lint else ""
            print(f"[{v.tier:9}] {v.id:10} {v.title[:48]:48} - {v.reason}{tag}")
        counts = {}
        for v in verdicts:
            counts[v.tier] = counts.get(v.tier, 0) + 1
        print("\nsummary: " + ", ".join(f"{k}={c}" for k, c in sorted(counts.items())))

    rejected = any(v.tier == "REJECTED" for v in verdicts)
    strict_fail = args.strict and any(v.tier == "CONFIRMED" and v.lint for v in verdicts)
    return 1 if (rejected or strict_fail) else 0


if __name__ == "__main__":
    raise SystemExit(main())
rea-analysis-optimizationmorluto/reaOptimize REA providers and shared analysis workflows for agent context, latency, CPU, and memory while preserving correctness and Evidence. Use for analysis bottlenecks and cache or query design; CI/test-suite optimization belongs to rea-ci-tests-audit.Stars: 471kUpdated Claude CodeCodexVerifiedInstall
rea-ci-tests-auditmorluto/reaAudit REA CI and test suites for correctness, fixture drift, false passes, and measured performance; use for suite or workflow investigations and efficiency changes, rather than ordinary feature testing or a single test failure.Stars: 471kUpdated Claude CodeCodexVerifiedInstall
rea-tool-designmorluto/reaAudit, investigate, design, or change REA investigation tools, CLI/MCP contracts, provider capabilities, and Evidence semantics.Stars: 471kUpdated Claude CodeCodexVerifiedInstall
ponytail-auditDietrichGebert/ponytailQuality audit of the whole repo: bugs, security, real load, missing tests, speed, and what to delete. Most important first.Stars: 194kUpdated Claude CodeCodexVerifiedInstall