hypnguyen1209/offensive-claude / skills/vulnerability-analysis
vulnerability-analysis
UnverifiedUse when auditing source code for vulnerabilities — drive CodeQL/Semgrep/Joern to taint untrusted data source-to-sink across injection, memory safety, deserialization/prototype-pollution, secrets/crypto/authz/race, and supply-chain risks
| File | Kind | Size |
|---|---|---|
| SKILL.md | skill md | 11.4 KB |
| references/deserialization-prototype-pollution.md | markdown | 8.3 KB |
| references/injection-source-patterns.md | markdown | 8.6 KB |
| references/memory-safety-c-cpp.md | markdown | 7.7 KB |
| references/secrets-crypto-authz-concurrency.md | markdown | 8.6 KB |
| references/supply-chain-dependency-audit.md | markdown | 7.3 KB |
| references/taint-engines-static-analysis.md | markdown | 11.0 KB |
| references/variant-hunting.md | markdown | 3.4 KB |
| scripts/dep_audit.py | script | 10.9 KB |
| scripts/deser_gadget_scan.py | script | 7.7 KB |
| scripts/evidence_kit.py | script | 13.0 KB |
| scripts/finding_dedup.py | script | 4.2 KB |
| scripts/joern_taint.sc | other | 3.6 KB |
| scripts/merge_runtime_evidence.py | script | 6.1 KB |
| scripts/path_conditions.py | script | 10.6 KB |
| scripts/provenance.py | script | 2.8 KB |
| scripts/sast_runner.py | script | 7.8 KB |
| scripts/secret_crypto_audit.py | script | 8.1 KB |
| scripts/taint_trace.py | script | 14.9 KB |
| scripts/validate_findings.py | script | 22.8 KB |
| scripts/variant_hunt.py | script | 11.8 KB |
| scripts/vocab_gate.py | script | 7.5 KB |
scripts/validate_findings.py
View on GitHub#!/usr/bin/env python3
"""validate_findings.py - evidence-grounding + false-positive harness for findings.
Enforces the CLAUDE.md mandate "assess real exploitability (not just pattern matching)"
mechanically, for LIVE/dynamic findings (the static-audit counterpart is
skills/references/taint-false-positive-reduction.md).
DESIGN: tiers are driven by STRUCTURED proof signals (explicit booleans), never by regex
over analyst prose. Prose paraphrase can flip a substring match in either direction
(a self-IDOR worded to dodge a kill phrase, or a real RCE rejected because it says "the
initial probe returned no output"). Booleans can't be paraphrased. The job of judging
whether a boolean is HONEST belongs to the adversarial finding-validator agent /
human review - this tool is the mechanical gate.
Two checks per finding:
1. GROUNDING - every cited evidence path must exist on disk and be non-empty.
2. CLASS PROOF - for a recognized vuln class, CONFIRMED requires the class's positive
proof signal to be explicitly true; an explicit disqualifier signal
forces REJECTED; otherwise the finding is POSSIBLE (gather more proof).
Output tiers: CONFIRMED | POSSIBLE | INFO | REJECTED (see finding-validation-runtime.md)
Finding input = JSON list. `proof` is an OBJECT of boolean signals:
[{"id":"F1","title":"SSRF in webhook fetch","cwe":"CWE-918","severity":"High",
"evidence":["logs/F1.txt"],
"proof":{"internal_response_read": true}}]
Per-class signals:
SSRF (918) confirm: internal_response_read
IDOR/BOLA (639/862/...) confirm: cross_identity_confirmed disqualify: self_idor
CORS (942) confirm: creds_reflected_origin
Open redirect (601) confirm: external_redirect disqualify: same_origin
XSS (79/80) confirm: script_executed disqualify: encoded_inert
RCE/cmd/deser (78/77/94/502) confirm: command_output_captured
Optional EVD-citation grounding (raptor-adoption): with --evidence-store evidence.json (an
evidence_kit store), any [EVD-XXX] a finding cites must EXIST and be VERIFIED, else the finding is
REJECTED (fail-closed - the mechanical half of finding-validator Q2/Q2.5). Hedge phrases
("probably", "should work", "likely exploitable") are linted as advisory warnings; --strict makes a
hedge/uncited warning on a CONFIRMED finding also fail the gate. Default behavior is unchanged.
CLI:
validate_findings.py --findings findings.json [--evidence ./evidence]
[--evidence-store evidence.json] [--strict] [--json]
exit 0 = clean; exit 1 = at least one REJECTED (or --strict lint fail); exit 2 = error
"""
from __future__ import annotations
import argparse
import json
import re
import sys
from dataclasses import dataclass, asdict, field
from pathlib import Path
from typing import Optional
sys.path.insert(0, str(Path(__file__).resolve().parent))
import provenance # noqa: E402 (sibling module: per-hop reachability provenance gate)
# Hedge phrases: ungrounded-language smell. ADVISORY only - they never drive a tier (prose is
# paraphrasable; only structured signals confirm). They surface "you wrote it like a guess" so a
# reviewer re-checks, and in --strict a CONFIRMED finding carrying them fails the lint gate.
HEDGE_PHRASES = [
r"\bprobably\b", r"\bshould\s+(?:be|work)\b", r"\blikely\s+exploitabl", r"\bmight\b",
r"\bmaybe\b", r"\bseems?\s+to\b", r"\bappears?\s+to\b", r"\bin\s+theory\b",
r"\bcould\s+(?:potentially|possibly)\b", r"\bi\s+(?:think|believe|assume)\b",
r"\btheoretically\b", r"\bpresumabl", r"\bguess", r"\bnot\s+sure\b",
]
_HEDGE_RX = [re.compile(p, re.I) for p in HEDGE_PHRASES]
_EVD_RX = re.compile(r"EVD-[A-Za-z0-9_.\-]+", re.I) # case-insensitive: 'evd-001' can't dodge the gate
# class -> cwe ids, title keywords (fallback class detection), confirm signal, disqualify signal
CLASS_RULES = [
{"name": "SSRF", "cwe": {"918"}, "kw": ["ssrf", "server-side request"],
"confirm": "internal_response_read", "disqualify": None,
"hint": "an internal/restricted response (not a bare DNS/OOB callback) was read back"},
{"name": "IDOR/BOLA", "cwe": {"639", "862", "566", "284", "1220"}, "kw": ["idor", "bola", "broken object", "broken access"],
"confirm": "cross_identity_confirmed", "disqualify": "self_idor",
"hint": "another principal's data was accessed using a second identity you control"},
{"name": "CORS", "cwe": {"942"}, "kw": ["cors", "cross-origin resource"],
"confirm": "creds_reflected_origin", "disqualify": None,
"hint": "ACAO reflected the origin AND Access-Control-Allow-Credentials:true, with a cross-origin authenticated read"},
{"name": "Open Redirect", "cwe": {"601"}, "kw": ["open redirect", "open-redirect"],
"confirm": "external_redirect", "disqualify": "same_origin",
"hint": "the final redirect Location host is external/attacker-controlled"},
{"name": "XSS", "cwe": {"79", "80"}, "kw": ["xss", "cross-site scripting"],
"confirm": "script_executed", "disqualify": "encoded_inert",
"hint": "script actually executed in the victim context (not reflected-but-encoded)"},
{"name": "RCE/Command Injection", "cwe": {"78", "77", "94", "502"}, "kw": ["rce", "command inj", "remote code", "deserial"],
"confirm": "command_output_captured", "disqualify": None,
"hint": "command output was captured (blind with no confirming oob is POSSIBLE, not CONFIRMED)"},
]
# Native memory-corruption classes need a machine-checkable REACHABILITY artifact to CONFIRM - a
# crash on paper is not demonstrated reachability. Without a gcov line-hit or a function trace the
# finding is POSSIBLE (the vuln line may never execute on the path you think). raptor-adoption F/F.
NATIVE_MEMORY_CWES = {
"119", "120", "121", "122", "124", "125", "126", "127", "129", "131", "134", "170",
"190", "191", "415", "416", "457", "476", "562", "590", "680", "763",
"786", "787", "788", "805", "822", "823", "824", "825", "843",
}
@dataclass
class Verdict:
id: str
title: str
tier: str # CONFIRMED | POSSIBLE | INFO | REJECTED
reason: str
cwe: str = ""
matched_class: Optional[str] = None
lint: list = field(default_factory=list) # advisory warnings (hedges, uncited) - not a tier
def to_dict(self) -> dict:
return asdict(self)
def _iter_strings(obj):
"""Yield every string reachable in a finding record - dict KEYS and values, list items - so a
citation/hedge can't hide in a key position (or any nested field) to dodge the scan."""
if isinstance(obj, str):
yield obj
elif isinstance(obj, dict):
for k, v in obj.items():
if isinstance(k, str):
yield k
yield from _iter_strings(v)
elif isinstance(obj, (list, tuple)):
for v in obj:
yield from _iter_strings(v)
def _cited_evds(f: dict) -> set:
cited = set()
for s in _iter_strings(f):
cited.update(m.upper() for m in _EVD_RX.findall(s)) # normalize case for comparison
return cited
def _lint_hedges(f: dict) -> list:
"""Hedge phrases anywhere in the finding's strings. Advisory: a confidence smell, never a
verdict. Scans every field (not just title/description) so a hedge in poc/recommendation/notes
still surfaces under --strict."""
hits = set()
for text in _iter_strings(f):
for rx in _HEDGE_RX:
m = rx.search(text)
if m:
hits.add(m.group(0).lower().strip())
return sorted(hits)
def _cwe_num(cwe) -> str:
m = re.search(r"(\d+)", str(cwe or ""))
return m.group(1) if m else ""
def _grounded(evidence, evidence_dir: Optional[Path]) -> tuple[bool, str]:
paths = evidence if isinstance(evidence, list) else ([evidence] if evidence else [])
if not paths:
return False, "finding cites no evidence artifact"
for p in paths:
if not isinstance(p, str):
return False, f"evidence entry is not a path string: {p!r}"
candidates = [Path(p)]
if evidence_dir:
candidates.append(evidence_dir / p)
ok = False
for c in candidates:
try:
if not (c.is_file() and c.stat().st_size > 0):
continue
if evidence_dir is not None and not _within(c, evidence_dir):
continue # outside the evidence dir -> not grounded
ok = True
break
except OSError:
continue
if not ok:
return False, f"evidence not found, empty, or outside the evidence dir: {p}"
return True, "all cited evidence present"
def _match_rule(cwe_num: str, title: str):
if cwe_num:
for rule in CLASS_RULES:
if cwe_num in rule["cwe"]:
return rule
t = (title or "").lower()
for rule in CLASS_RULES:
if any(k in t for k in rule["kw"]):
return rule
return None
def _within(child: Path, root: Path) -> bool:
try:
rc, rr = child.resolve(), root.resolve()
except OSError:
return False
return rc == rr or rr in rc.parents
def _resolve(p, evidence_dir: Optional[Path]):
"""Resolve an evidence/proof path. When evidence_dir is set, a candidate is accepted ONLY if it
lives inside it - an absolute path or '..' escape outside the evidence dir is rejected, so every
proof artifact is tied to the engagement's evidence set."""
if not isinstance(p, str) or not p:
return None
cands = [Path(p)] + ([evidence_dir / p] if evidence_dir else [])
for c in cands:
try:
if not (c.is_file() and c.stat().st_size > 0):
continue
if evidence_dir is not None and not _within(c, evidence_dir):
continue # outside the evidence dir -> not grounded
return c
except OSError:
continue
return None
def _check_gcov(cov: dict, evidence_dir: Optional[Path]) -> tuple:
"""A gcov line-hit proof: the vulnerable source line is NOT '#####' (never executed)."""
path, line = cov.get("gcov"), cov.get("line")
if not isinstance(line, int):
return False, "coverage_proof.line must be an integer"
fpath = _resolve(path, evidence_dir)
if fpath is None:
return False, f"coverage_proof gcov not found/empty: {path!r}"
try:
text = fpath.read_text(encoding="utf-8", errors="replace")
except OSError:
return False, f"coverage_proof gcov unreadable: {path!r}"
for raw in text.splitlines():
parts = raw.split(":", 2) # gcov format: "<count>:<lineno>:<source>"
if len(parts) < 2:
continue
count_s, lineno_s = parts[0].strip(), parts[1].strip()
if lineno_s == str(line):
# ALLOWLIST: a real gcov execution count is a positive integer. Reject everything else -
# '#####'/'-'/'=====' (never executed), '0' (zero executions), and any forged/garbage
# token ('exec', '-5', 'banana') that a blocklist would have let through (fail-open).
shown = count_s
count_s = count_s.replace(",", "").rstrip("*") # thousands sep; '*' = executed, has a 0-count block
# ASCII decimal ONLY (re.fullmatch r"[0-9]+", not str.isdigit which accepts fullwidth/
# Arabic-Indic Unicode digits) and strictly positive.
if re.fullmatch(r"[0-9]+", count_s) and int(count_s) > 0:
return True, f"gcov shows line {line} executed (count={count_s})"
return False, f"gcov line {line} not executed (count={shown!r}; need a positive integer hit)"
return False, f"line {line} not present in gcov {path!r}"
def _check_trace(tr: dict, evidence_dir: Optional[Path]) -> tuple:
"""A function-trace proof: the vulnerable function name appears in a runtime trace log."""
path, fn = tr.get("log"), tr.get("function")
if not isinstance(fn, str) or not fn:
return False, "trace_proof.function missing"
fpath = _resolve(path, evidence_dir)
if fpath is None:
return False, f"trace_proof log not found/empty: {path!r}"
try:
text = fpath.read_text(encoding="utf-8", errors="replace")
except OSError:
return False, f"trace_proof log unreadable: {path!r}"
# A trace proof must evidence EXECUTION, not a textual mention. We require a STRUCTURAL call
# record - the function with an argument paren `fn(` (ltrace/uftrace/frame), a gdb/rr backtrace
# frame `#N ... fn`, or a trace arrow `-> fn` / `>>> fn`. We do NOT match English keywords like
# "call"/"enter" as substrings: those appear in prose ("never called", "recall", "ENTERPRISE",
# "do not invoke fn") and would fail open. Comment/prose lines (#, //, *, ;) are skipped.
fnre = re.escape(fn)
word = re.compile(r"\b" + fnre + r"\b")
call = re.compile(r"\b" + fnre + r"\s*\(") # fn( ... ) -> a call / frame
arrow = re.compile(r"(?:>>>|->|=>)\s*" + fnre + r"\b") # uftrace / trace arrow into fn
frame = re.compile(r"^#\d+\b.*\b" + fnre + r"\b") # gdb/rr backtrace frame (authoritative)
for raw in text.splitlines():
line = raw.strip()
if not word.search(line):
continue
if frame.match(line):
return True, f"trace shows {fn!r} in a stack frame"
if line[:2] in ("//", "/*") or line[:1] in ("#", ";", "*"):
continue # comment/prose line - not an exec record
if call.search(line) or arrow.search(line):
return True, f"trace shows {fn!r} in an execution record (call/frame/arrow)"
return False, (f"function {fn!r} not found in an EXECUTION context in {path!r} - needs a call "
"'fn(...)', a stack frame '#N ... fn', or a trace arrow; a bare mention in "
"prose/comment/symbol-dump is not reachability")
def _native_reachability(signals: dict, evidence_dir: Optional[Path]) -> tuple:
"""Native memory-corruption needs a reachability artifact: a gcov line-hit OR a function trace.
Returns (ok, why). Fail-closed: a malformed/missing artifact is NOT reachability."""
cov = signals.get("coverage_proof")
cov_reason = "no coverage_proof"
if isinstance(cov, dict):
ok, why = _check_gcov(cov, evidence_dir)
if ok:
return True, why
cov_reason = why
tr = signals.get("trace_proof")
tr_reason = "no trace_proof"
if isinstance(tr, dict):
ok, why = _check_trace(tr, evidence_dir)
if ok:
return True, why
tr_reason = why
return False, (f"no demonstrated reachability ({cov_reason}; {tr_reason}) - provide "
"proof.coverage_proof={gcov,line} with a non-'#####' hit, or "
"proof.trace_proof={log,function} naming the executed vulnerable function")
def _apply_provenance_cap(verdict: Verdict, f: dict) -> Verdict:
"""PR-3/T2 per-hop provenance gate. If a finding declares a `reachability` path, a CONFIRMED
verdict is capped to POSSIBLE unless EVERY hop is EXTRACTED (graphify weakest-hop rule). This is
fail-closed (unknown/missing per-hop provenance => AMBIGUOUS) and there is NO promotion path -
a reachability claim can only *lower* the tier, never raise it."""
reach = f.get("reachability")
if reach is None:
return verdict # no path declared -> unchanged (live findings, etc.)
if verdict.tier != "CONFIRMED":
return verdict # cap only ever downgrades; never touches POSSIBLE/INFO/REJECTED
if provenance.all_extracted(reach):
return verdict # all-EXTRACTED path -> CONFIRMED stands
weakest = provenance.weakest(reach)
verdict.tier = "POSSIBLE"
verdict.reason = (f"reachability weakest hop is {weakest} (not all-EXTRACTED) -> capped to "
f"POSSIBLE; {verdict.reason}")
return verdict
def evaluate_finding(f: dict, evidence_dir: Optional[Path], *,
evidence_ids: Optional[set] = None,
verified_ids: Optional[set] = None) -> Verdict:
"""Public entry: run the mechanical class/grounding gate, then apply the provenance cap."""
verdict = _evaluate_core(f, evidence_dir, evidence_ids=evidence_ids, verified_ids=verified_ids)
if isinstance(f, dict):
verdict = _apply_provenance_cap(verdict, f)
return verdict
def _evaluate_core(f: dict, evidence_dir: Optional[Path], *,
evidence_ids: Optional[set] = None,
verified_ids: Optional[set] = None) -> Verdict:
if not isinstance(f, dict): # a malformed (non-object) element must not crash the run
return Verdict("?", "", "REJECTED", f"malformed finding: not a JSON object ({type(f).__name__})", "")
fid = str(f.get("id", "?"))
title = str(f.get("title", ""))
cwe = str(f.get("cwe", ""))
cwe_num = _cwe_num(cwe)
severity = str(f.get("severity", "")).lower()
proof = f.get("proof")
signals = proof if isinstance(proof, dict) else {}
lint = [f"hedge: '{h}'" for h in _lint_hedges(f)]
ok, why = _grounded(f.get("evidence"), evidence_dir)
if not ok:
return Verdict(fid, title, "REJECTED", f"ungrounded: {why}", cwe, lint=lint)
# EVD-citation grounding (only when an evidence store was supplied). A cited evidence id that
# is unknown (dangling) or not VERIFIED breaks grounding -> REJECTED, fail-closed. This is the
# mechanical half of finding-validator Q2/Q2.5 (grounded + authentic at source).
if evidence_ids is not None:
cited = _cited_evds(f)
dangling = sorted(cited - evidence_ids)
if dangling:
return Verdict(fid, title, "REJECTED",
f"cites unknown evidence id(s): {', '.join(dangling)}", cwe, lint=lint)
if verified_ids is not None:
unverified = sorted(cited - verified_ids)
if unverified:
return Verdict(fid, title, "REJECTED",
f"cites non-VERIFIED evidence id(s): {', '.join(unverified)} "
"(run evidence_kit.py verify)", cwe, lint=lint)
if not cited:
lint.append("uncited: no [EVD-XXX] citation despite an evidence store")
rule = _match_rule(cwe_num, title)
if rule:
if rule["disqualify"] and signals.get(rule["disqualify"]) is True:
return Verdict(fid, title, "REJECTED",
f"{rule['name']} disqualified: proof.{rule['disqualify']} is true", cwe, rule["name"], lint)
if signals.get(rule["confirm"]) is True:
if severity in ("info", "informational"):
return Verdict(fid, title, "INFO", f"{rule['name']} proven but informational", cwe, rule["name"], lint)
return Verdict(fid, title, "CONFIRMED",
f"{rule['name']} impact proven via proof.{rule['confirm']} and grounded", cwe, rule["name"], lint)
kind = "set" if isinstance(proof, dict) else "provide a structured proof object and set"
return Verdict(fid, title, "POSSIBLE",
f"{rule['name']}: not confirmed - {kind} proof.{rule['confirm']}=true once {rule['hint']}",
cwe, rule["name"], lint)
# Native memory-corruption: a crash on paper is not exploitability - require a reachability artifact.
if cwe_num in NATIVE_MEMORY_CWES:
ok, why = _native_reachability(signals, evidence_dir)
if not ok:
return Verdict(fid, title, "POSSIBLE", f"native memory-corruption: {why}",
cwe, "Native Memory-Corruption", lint)
if severity in ("info", "informational"):
return Verdict(fid, title, "INFO", f"native bug reachable ({why}) but informational",
cwe, "Native Memory-Corruption", lint)
return Verdict(fid, title, "CONFIRMED",
f"native memory-corruption with demonstrated reachability: {why}",
cwe, "Native Memory-Corruption", lint)
if severity in ("info", "informational"):
return Verdict(fid, title, "INFO", "grounded; informational severity", cwe, lint=lint)
return Verdict(fid, title, "CONFIRMED", "grounded; no automated class check applies (reviewer must confirm impact)", cwe, lint=lint)
def load_findings(path: str) -> list[dict]:
with open(path, "r", encoding="utf-8") as fh:
data = json.load(fh)
if isinstance(data, dict):
data = data.get("findings", [])
if not isinstance(data, list):
raise ValueError("findings file must be a JSON list (or {findings:[...]})")
return data
def main(argv: Optional[list[str]] = None) -> int:
p = argparse.ArgumentParser(description="Evidence-grounding + false-positive harness for findings.")
p.add_argument("--findings", required=True, help="JSON list of finding records")
p.add_argument("--evidence", help="directory holding evidence artifacts (paths resolved against it)")
p.add_argument("--evidence-store", help="evidence_kit JSON; enables [EVD-XXX] citation grounding")
p.add_argument("--strict", action="store_true",
help="lint warnings (hedges / uncited) on a CONFIRMED finding also fail the gate")
p.add_argument("--json", action="store_true")
args = p.parse_args(argv)
try:
findings = load_findings(args.findings)
except (OSError, ValueError, json.JSONDecodeError) as exc:
print(f"error: {exc}", file=sys.stderr)
return 2
evidence_ids = verified_ids = None
if args.evidence_store:
try:
import evidence_kit
store = evidence_kit.EvidenceStore.load(args.evidence_store)
store.verify_all() # re-verify integrity before trusting citations
evidence_ids = {i.upper() for i in store.ids()} # compare case-insensitively
verified_ids = {i.upper() for i in store.verified_ids()}
except Exception as exc: # a broken store must not silently disable the check
print(f"error: --evidence-store: {exc}", file=sys.stderr)
return 2
ev_dir = Path(args.evidence) if args.evidence else None
verdicts = [evaluate_finding(f, ev_dir, evidence_ids=evidence_ids, verified_ids=verified_ids)
for f in findings]
if args.json:
print(json.dumps([v.to_dict() for v in verdicts], indent=2))
else:
for v in verdicts:
tag = f" [lint: {'; '.join(v.lint)}]" if v.lint else ""
print(f"[{v.tier:9}] {v.id:10} {v.title[:48]:48} - {v.reason}{tag}")
counts = {}
for v in verdicts:
counts[v.tier] = counts.get(v.tier, 0) + 1
print("\nsummary: " + ", ".join(f"{k}={c}" for k, c in sorted(counts.items())))
rejected = any(v.tier == "REJECTED" for v in verdicts)
strict_fail = args.strict and any(v.tier == "CONFIRMED" and v.lint for v in verdicts)
return 1 if (rejected or strict_fail) else 0
if __name__ == "__main__":
raise SystemExit(main())
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| rea-analysis-optimizationmorluto/reaOptimize REA providers and shared analysis workflows for agent context, latency, CPU, and memory while preserving correctness and Evidence. Use for analysis bottlenecks and cache or query design; CI/test-suite optimization belongs to rea-ci-tests-audit. | Stars: 471k | Updated | Claude CodeCodex | Verified | Install |
| rea-ci-tests-auditmorluto/reaAudit REA CI and test suites for correctness, fixture drift, false passes, and measured performance; use for suite or workflow investigations and efficiency changes, rather than ordinary feature testing or a single test failure. | Stars: 471k | Updated | Claude CodeCodex | Verified | Install |
| rea-tool-designmorluto/reaAudit, investigate, design, or change REA investigation tools, CLI/MCP contracts, provider capabilities, and Evidence semantics. | Stars: 471k | Updated | Claude CodeCodex | Verified | Install |
| ponytail-auditDietrichGebert/ponytailQuality audit of the whole repo: bugs, security, real load, missing tests, speed, and what to delete. Most important first. | Stars: 194k | Updated | Claude CodeCodex | Verified | Install |