Skip to content

alexgreensh/repo-forensics / skills/repo-forensics

repo-forensics

Flagged

Security forensics for git repos, AI skills, and MCP servers. Audits dependencies, detects prompt injection, credential theft, runtime dynamism, manifest drift, known CVEs, CISA KEV (actively exploited) vulns, and 2026 attack patterns. Not for fixing vulnerabilities or pentesting.

FileKindSize
.gitignoreother193 B
SKILL.mdskill md43.4 KB
checksums.jsonconfig10.8 KB
checksums.json.sigother64 B
data/README.mdmarkdown1.3 KB
data/compromised_versions.jsonconfig188.3 KB
data/rule_ids.csvother35.3 KB
data/rulepacks/.gitkeepother221 B
data/rulepacks/dead_anchors.jsonconfig9.0 KB
data/rulepacks/mcp_security.jsonconfig31.2 KB
data/rulepacks/runtime_dynamism.jsonconfig46.7 KB
data/rulepacks/sast.jsonconfig107.6 KB
data/rulepacks/secrets.jsonconfig30.4 KB
data/rulepacks/shared.jsonconfig4.4 KB
data/rulepacks/skill_threats.jsonconfig95.4 KB
data/runtime_floors.jsonconfig2.3 KB
data/yara/cryptominers.yarother1.8 KB
data/yara/hacktools.yarother2.2 KB
data/yara/malware.yarother2.2 KB
data/yara/manifest.jsonconfig8.0 KB
data/yara/webshells.yarother2.5 KB
references/cursor-hook-contract.mdmarkdown5.6 KB
references/kernel_privesc_sandbox_pack.mdmarkdown5.0 KB
references/mcp-attack-patterns.mdmarkdown8.7 KB
references/research_sources.mdmarkdown8.0 KB
references/threat_patterns.mdmarkdown20.3 KB
scripts/_context_gate.pyscript21.7 KB
scripts/_ed25519.pyscript5.1 KB
scripts/_pyc_unmarshal.pyscript17.1 KB
scripts/_shared_patterns.pyscript4.8 KB
scripts/adjudication.pyscript16.4 KB
scripts/adjudication_bridge.pyscript1.0 KB
scripts/aggregate_json.pyscript69.2 KB
scripts/auto_scan.pyexecutablescript152.1 KB
scripts/corpus_sync.pyscript10.0 KB
scripts/cursor_pre_scan.pyscript22.5 KB
scripts/dast_sandbox.sbother1.6 KB
scripts/dead_anchors_extract.pyscript14.8 KB
scripts/dead_anchors_probe.pyscript27.1 KB
scripts/forensics_core.pyscript176.5 KB
scripts/gen_rule_ids.pyscript8.6 KB
scripts/hook_adapter.pyscript19.4 KB
scripts/ioc_manager.pyscript49.6 KB
scripts/parse_pnpm_lock.pyscript7.9 KB
scripts/pre_scan.pyscript143.5 KB
scripts/pypi_pins.pyscript8.1 KB
scripts/refresh_controller.pyexecutablescript48.4 KB
scripts/refresh_threat_dbs.pyexecutablescript19.4 KB
scripts/rule_loader.pyscript44.8 KB
scripts/rulepack_feed.pyscript26.4 KB
scripts/run_forensics.shexecutablescript20.3 KB
scripts/sarif_export.pyscript18.6 KB
scripts/scan_agent_skills.pyscript33.2 KB
scripts/scan_archive.pyscript29.1 KB
scripts/scan_ast.pyscript44.9 KB
scripts/scan_binary.pyscript10.9 KB
scripts/scan_bytecode.pyscript28.2 KB
scripts/scan_dast.pyscript28.8 KB
scripts/scan_dataflow.pyscript15.4 KB
scripts/scan_dead_anchors.pyscript15.9 KB
scripts/scan_decode.pyscript37.5 KB
scripts/scan_dependencies.pyscript124.5 KB
scripts/scan_devcontainer.pyscript8.5 KB
scripts/scan_entropy.pyscript15.6 KB
scripts/scan_entrypoint.pyscript34.3 KB
scripts/scan_git_config.pyscript96.3 KB
scripts/scan_git_forensics.pyscript24.1 KB
scripts/scan_history.pyscript14.5 KB
scripts/scan_infra.pyscript54.3 KB
scripts/scan_integrity.pyscript41.0 KB
scripts/scan_lifecycle.pyscript65.1 KB
scripts/scan_manifest_drift.pyscript47.0 KB
scripts/scan_mcp_security.pyscript42.3 KB
scripts/scan_oversize.pyscript11.7 KB
scripts/scan_post_incident.pyscript36.9 KB
scripts/scan_provenance.pyscript19.4 KB
scripts/scan_runtime_dynamism.pyscript23.5 KB
scripts/scan_runtime_floors.pyscript33.5 KB
scripts/scan_sast.pyscript59.1 KB
scripts/scan_secrets.pyscript11.9 KB
scripts/scan_skill_threats.pyscript102.4 KB
scripts/scan_splitstream.pyscript29.5 KB
scripts/scan_yara.pyscript19.7 KB
scripts/session_scan.pyscript54.1 KB
scripts/suppressions.pyscript15.4 KB
scripts/validate_manifests.pyscript13.7 KB
scripts/verify_install.pyscript26.2 KB
scripts/vuln_feed.pyscript31.1 KB
tests/conftest.pyscript13.4 KB
tests/corpus/DEAD_ANCHORS_CORPUS.mdmarkdown2.3 KB
tests/corpus/budgets.jsonconfig5.6 KB
tests/cursor_helpers.pyscript12.6 KB
tests/fixtures/legacy_main_blocks.jsonconfig10.3 KB
tests/fixtures/powershell_verdict_corpus.jsonconfig114.9 KB
tests/golden/parity_mcp_security.jsonconfig17.5 KB
tests/golden/parity_runtime_dynamism.jsonconfig31.7 KB
tests/golden/parity_sast.jsonconfig59.8 KB
tests/golden/parity_secrets.jsonconfig18.8 KB
tests/golden/parity_skill_threats.jsonconfig48.8 KB
tests/parity_corpus.pyscript6.3 KB
tests/powershell_hook.pyscript8.2 KB
tests/report_builders.pyscript5.0 KB
tests/schemas/sarif-schema-2.1.0.jsonconfig109.1 KB
tests/shell_compat.pyscript2.2 KB
tests/test_adjudication_bridge.pyscript832 B
tests/test_adjudication_containment.pyscript1.4 KB
tests/test_adjudication_output.pyscript20.0 KB
tests/test_adjudication_suppressions.pyscript18.7 KB
tests/test_advisory_adjudication.pyscript1.3 KB
tests/test_agent_relay_rules.pyscript7.8 KB
tests/test_aggregate_json.pyscript10.0 KB
tests/test_auto_scan_patterns.pyscript10.5 KB
tests/test_benign_corpus.pyscript16.4 KB
tests/test_benign_new_scanners.pyscript6.1 KB
tests/test_build_output_skip.pyscript10.6 KB
tests/test_capability_ledger.pyscript18.8 KB
tests/test_codex_install.pyscript10.0 KB
tests/test_codex_marketplace.pyscript3.3 KB
tests/test_comment_false_positives.pyscript2.2 KB
tests/test_composer_dependencies.pyscript4.0 KB
tests/test_compromised_versions.pyscript51.2 KB
tests/test_context_gate.pyscript32.4 KB
tests/test_coverage_status.pyscript7.6 KB
tests/test_css_compounding_shrink.pyscript20.8 KB
tests/test_cursor_adapter.pyscript7.8 KB
tests/test_cursor_benign.pyscript3.4 KB
tests/test_cursor_degrade.pyscript12.7 KB
tests/test_cursor_install.pyscript17.1 KB
tests/test_cursor_latency.pyscript7.2 KB
tests/test_cursor_parity.pyscript5.8 KB
tests/test_cursor_redteam_fn.pyscript4.9 KB
tests/test_cursor_tamper_modify.pyscript14.2 KB
tests/test_cursor_wiring.pyscript32.3 KB
tests/test_d1_yara_correlation.pyscript9.2 KB
tests/test_dead_anchors_extract.pyscript7.6 KB
tests/test_dead_anchors_probe.pyscript19.7 KB
tests/test_evidence_engine_parity.pyscript8.1 KB
tests/test_evidence_model.pyscript11.4 KB
tests/test_forensics_core.pyscript79.2 KB
tests/test_fp_hardening_r3.pyscript9.7 KB
tests/test_fp_hardening_r4.pyscript18.8 KB
tests/test_fp_regression_pairs.pyscript62.5 KB
tests/test_fp_structural.pyscript21.2 KB
tests/test_freshness.pyscript6.2 KB
tests/test_frontmatter_validity.pyscript7.0 KB
tests/test_gpg_rce_hotfix.pyscript4.8 KB
tests/test_hook_inert_data.pyscript6.1 KB
tests/test_hook_scripts.pyscript14.0 KB
tests/test_inspection_cpu_budget.pyscript11.6 KB
tests/test_ioc_manager.pyscript55.4 KB
tests/test_kernel_privesc_rules.pyscript27.8 KB
tests/test_legit_precision_regressions.pyscript6.8 KB
tests/test_lethal_trifecta.pyscript20.7 KB
tests/test_manifest_signature.pyscript1.5 KB
tests/test_manifest_validity.pyscript2.9 KB
tests/test_monotonic_trust_e2e.pyscript4.4 KB
tests/test_non_breaking_contract.pyscript22.7 KB
tests/test_openclaw_install.pyscript6.6 KB
tests/test_pack_parity.pyscript24.8 KB
tests/test_parse_pnpm_lock.pyscript10.7 KB
tests/test_plugin_checkout_provenance.pyscript11.1 KB
tests/test_powershell_hook_wire.pyscript10.6 KB
tests/test_powershell_verdict_corpus.pyscript6.2 KB
tests/test_pr44_hardening.pyscript6.5 KB
tests/test_pre_scan.pyscript19.6 KB
tests/test_pre_scan_composer.pyscript7.4 KB
tests/test_precondition_fields.pyscript4.6 KB
tests/test_provenance_offline.pyscript2.5 KB
tests/test_published_rulepack_feed_current.pyscript1.4 KB
tests/test_py38_compat.pyscript3.3 KB
tests/test_pypi_pinned_versions.pyscript19.3 KB
tests/test_pyproject_string_forms.pyscript2.2 KB
tests/test_redteam_fn_regression.pyscript32.5 KB
tests/test_reflective_fetch_execute.pyscript23.5 KB
tests/test_reflective_review_fixes.pyscript17.8 KB
tests/test_refresh_controller.pyscript19.9 KB
tests/test_refresh_rulepacks.pyscript12.4 KB
tests/test_review_gate_regressions.pyscript9.3 KB
tests/test_rule_loader.pyscript23.2 KB
tests/test_rulepack_feed.pyscript38.1 KB
tests/test_run_forensics_json.pyscript1012 B
tests/test_run_scanner_fail_loud.pyscript6.5 KB
tests/test_runtime_floors.pyscript19.8 KB
tests/test_sandbox_probing_rules.pyscript9.6 KB
tests/test_sarif_export.pyscript19.9 KB
tests/test_scan_agent_skills.pyscript31.3 KB
tests/test_scan_archive.pyscript11.6 KB
tests/test_scan_ast.pyscript12.8 KB
tests/test_scan_binary.pyscript15.3 KB
tests/test_scan_bytecode.pyscript21.5 KB
tests/test_scan_dast.pyscript32.1 KB
tests/test_scan_dataflow.pyscript7.1 KB
tests/test_scan_dead_anchors.pyscript11.5 KB
tests/test_scan_decode.pyscript28.0 KB
tests/test_scan_dependencies.pyscript37.8 KB
tests/test_scan_devcontainer.pyscript8.5 KB
tests/test_scan_entropy.pyscript11.2 KB
tests/test_scan_entrypoint.pyscript19.5 KB
tests/test_scan_git_config.pyscript58.5 KB
tests/test_scan_git_config_review_fixes.pyscript33.3 KB

scripts/scan_secrets.py

View on GitHub
#!/usr/bin/env python3
"""
scan_secrets.py - Secret Scanner (rules-as-data)

Detects hardcoded API keys, tokens, certificates, private keys, database
credentials, and generic secret assignments. As of v2.10 the detection
patterns live in a JSON rule pack (data/rulepacks/secrets.json), loaded at
module import via rule_loader. The pack is the single source of truth; there
is no hardcoded fallback table. If the pack cannot be loaded (a corrupted or
tampered install), the scanner emits one loud diagnostic finding and scans no
patterns rather than carrying a stale 500-line copy of the rules in code (a
corrupted install is independently caught by the integrity scanner).

The .env-variant allowlist, the MAX_LINE_LENGTH skip, and binary-file skipping
are scanning *context* machinery, not detection rules, so they stay in code.

Created by Alex Greenshpun
"""

import os
import re
import sys

sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
import forensics_core as core
import rule_loader

SCANNER_NAME = "secrets"

ENV_FILE_VARIANTS = {
    '.env', '.env.local', '.env.production', '.env.staging',
    '.env.development', '.env.test', '.env.docker', '.env.container',
}
ENV_FILE_SAFE = {'.env.example', '.env.template', '.env.sample'}

# --- Known documentation and regression-test placeholders ---
# Explicitly fake values that have no credential value. The scanner does
# NOT flag them as real secrets. Suppression is at the scanner layer (not the
# regex) so the rulepack examples still accurately document what the regex
# matches. All checks are EXACT full-string matches on the regex-captured
# snippet, never substring searches, so a real credential in a URI whose host
# contains "example.com" (e.g. db.example.com) is NOT suppressed.
_DOC_PLACEHOLDER_EXACT = frozenset({
    "AKIAIOSFODNN7EXAMPLE",  # canonical example AWS key ID
    "AKIA1234567890123456",  # sequential dummy key used in redaction tests
    "ghp_0123456789abcdefghijklmnopqrstuvwxyz",  # canonical example GitHub PAT
    "AKIA1234567890ABCDEF",  # ordered digits/hex in credential-preservation test
    "sk-ant-BearerSecretTOKEN1234567890abcdefXYZ",  # literal BearerSecret test marker
    "ghp_ABCDEFghijklmnopqrstuvwxyz0123456789",  # ordered alphabet benchmark sample
    "ghp_aBcDeFgHiJkLmNoPqRsTuVwXyZ0123456789",  # alternating-case benchmark sample
    "ghp_abcdefghijklmnopqrstuvwxyz0123456789",  # ordered alphabet benchmark sample
    "ghp_abcdefghijklmnopqrstuvwxyz0123456789AB",  # full traceback benchmark sample
    "AIzaSyBcDeFgHiJkLmNoPqRsTuVwXyZ01234567",  # ordered alphabet benchmark sample
    "AIzaSyBcDeFgHiJkLmNoPqRsTuVwXyZ0123456789abc",  # full log benchmark sample
})

# Canonical example JWT from jwt.io. The FULL token (header.payload.signature)
# is the placeholder. Only an exact match on the complete token suppresses; a
# token that shares the header.payload but has a different (real) signature is
# NOT suppressed, because the signature is the secret-bearing segment.
_JWT_IO_SAMPLE_FULL = (
    "eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9."
    "eyJzdWIiOiIxMjM0NTY3ODkwIn0."
    "SflKxwRJSMeKKF2QT4fwpMeJf36POk6yJV_adQssw5c"
)


def _is_doc_placeholder(snippet, full_line="", match_start=None, match_end=None):
    """Return True if the matched snippet is a known placeholder
    that must not be flagged as a real secret.

    Checks are exact matches on a complete token, never substring searches.
    Some regexes capture a fixed-length prefix, so compare the whole source
    token before accepting a placeholder. A real credential in a URI
    whose host contains 'example.com' (e.g. db.example.com) is NOT suppressed
    because the full URI string does not match any known placeholder.
    """
    if match_start is not None and re.fullmatch(r"[A-Za-z0-9_-]+", snippet):
        escaped_bell = full_line[:match_start].endswith(r"\x07")
        if (match_start > 0 and not escaped_bell
                and re.match(r"[A-Za-z0-9_-]", full_line[match_start - 1])):
            return False
        full_token = re.match(r"[A-Za-z0-9_-]+", full_line[match_start:])
        return bool(full_token and full_token.group(0) in _DOC_PLACEHOLDER_EXACT)
    if match_end is not None and match_end < len(full_line) and re.match(r"[A-Za-z0-9_-]", full_line[match_end]):
        return False
    if snippet in _DOC_PLACEHOLDER_EXACT:
        return True
    if snippet == _JWT_IO_SAMPLE_FULL:
        return True
    return False


_SVG_GEOMETRY_ATTR = re.compile(
    r'(?<![\w:-])(?:d|points|viewBox|transform)\s*=\s*"([0-9eE.,\s+\-]*(?:[MmLlHhVvCcSsQqTtAaZz][0-9eE.,\s+\-]*)*)"')


def _ip_match_is_noise(line, match, rel_path):
    """True when a SC-NET-001 hit is not an address at all: a four-number slice
    of a longer dotted number run (version strings, numeric tables), or
    zero-padded numbers inside SVG geometry attributes (path coordinates, not octets).
    Loopback and unspecified addresses are real addresses and are kept as-is."""
    text = match.group(0)
    before = line[max(0, match.start() - 2):match.start()]
    after = line[match.end():match.end() + 2]
    if re.search(r'\d\.$', before) or re.match(r'\.\d', after):
        return True
    ext = os.path.splitext(rel_path)[1].lower()
    if ext == '.svg' and any(len(o) > 1 and o[0] == '0' for o in text.split('.')):
        # only when the numbers sit inside a geometry attribute value; the
        # same digits in script text, an href or element text are kept.
        for attr in _SVG_GEOMETRY_ATTR.finditer(line):
            if attr.start(1) <= match.start() and match.end() <= attr.end(1):
                return True
    return False


def _emit_line_findings(findings, rule, line, line_no, rel_path):
    """Emit one finding per non-placeholder match of `rule` on `line`.

    Iterates EVERY match (finditer, not just the leftmost search) so a
    canonical placeholder earlier on a line cannot suppress a real key later
    on the same line, and two real keys on one line are both reported. Each
    individual match that is exactly a placeholder is skipped; the remaining
    matches are still examined. (Previously a single rule.regex.search per
    line returned only the first match, so a placeholder first match silently
    dropped every later real key on that line.)
    """
    for match in rule.regex.finditer(line):
        snippet = match.group(0)
        if _is_doc_placeholder(snippet, full_line=line, match_start=match.start(), match_end=match.end()):
            continue
        if rule.id == 'SC-NET-001' and _ip_match_is_noise(line, match, rel_path):
            continue
        if len(snippet) > 80:
            snippet = snippet[:77] + "..."
        findings.append(core.Finding(
            scanner=SCANNER_NAME,
            severity=rule.severity,
            title=rule.title,
            description="Potential hardcoded secret detected",
            file=rel_path,
            line=line_no,
            snippet=snippet,
            category="secret",
            rule_id=rule.id,
            confidence=rule.confidence,
            attacker=rule.attacker,
            boundary=rule.boundary,
            asset=rule.asset,
        ))


# Detection rules load from the shipped pack at import time (rule_loader
# memoizes, so this parses once per process). load_pack returns None only when
# the pack file is missing or schema-incompatible -> a corrupted/tampered
# install, surfaced as PACK_LOAD_ERROR below.
_PACK = rule_loader.load_pack(SCANNER_NAME)
PACK_LOAD_ERROR = _PACK is None
# Flat list of compiled rules (secrets has no extension gating; every rule
# applies to every text line).
_RULES = _PACK.all_rules if _PACK is not None else []

# B6 fix: emit the pack-load-failure diagnostic exactly ONCE per scanner run,
# not once per scanned file (which could flood a large repo with thousands of
# duplicate criticals and cause OOM in the aggregator).
_pack_error_emitted = False


def _pack_load_finding(rel_path):
    """The single loud diagnostic emitted (once per scanner run) when the
    rule pack failed to load. Critical so it cannot be missed; the operator is
    told to reinstall. We deliberately do NOT fall back to a hardcoded copy."""
    return core.Finding(
        scanner=SCANNER_NAME, severity="critical",
        title="Secret rule pack failed to load",
        description=("data/rulepacks/secrets.json is missing or "
                     "schema-incompatible; secret scanning is disabled. "
                     "Reinstall repo-forensics to restore detection."),
        file=rel_path, line=0,
        snippet="rule pack failed to load",
        category="scanner-integrity",
    )


def scan_file(file_path, rel_path):
    global _pack_error_emitted
    findings = []

    if PACK_LOAD_ERROR:
        # B6: emit the diagnostic only on the first call; subsequent files get
        # an empty list so the aggregator is not flooded with duplicates.
        if not _pack_error_emitted:
            _pack_error_emitted = True
            return [_pack_load_finding(rel_path)]
        return []

    basename = os.path.basename(file_path)
    if basename in ENV_FILE_VARIANTS and basename not in ENV_FILE_SAFE:
        findings.append(core.Finding(
            scanner=SCANNER_NAME, severity="high",
            title=f"Unencrypted {basename} File in Repository",
            description=f"{basename} likely contains plaintext secrets and should be in .gitignore",
            file=rel_path, line=0,
            snippet=f"{basename} found in repo (should never be committed)",
            category="secret-storage"
        ))

    try:
        with open(file_path, 'r', encoding='utf-8', errors='ignore') as f:
            lines = f.readlines()

        for i, line in enumerate(lines):
            # Skip mega-lines to prevent O(n^2) regex backtracking
            line = core.clip_line(line)

            for rule in _RULES:
                _emit_line_findings(findings, rule, line, i + 1, rel_path)
    except (OSError, UnicodeDecodeError) as e:
        print(f"[!] Skipped {rel_path}: {e}", file=sys.stderr)
    return findings


def scan_text(text, rel_path):
    """Secret-detection over an in-memory text blob — an extracted archive
    member. KTD7 in-memory entry point scan_archive recurses. Mirrors
    scan_file's per-rule emission; the env-file basename check is applied from
    rel_path's basename so a `.env` packed inside an archive is still flagged."""
    global _pack_error_emitted
    findings = []
    if PACK_LOAD_ERROR:
        if not _pack_error_emitted:
            _pack_error_emitted = True
            return [_pack_load_finding(rel_path)]
        return []

    basename = os.path.basename(rel_path)
    if basename in ENV_FILE_VARIANTS and basename not in ENV_FILE_SAFE:
        findings.append(core.Finding(
            scanner=SCANNER_NAME, severity="high",
            title=f"Unencrypted {basename} File in Repository",
            description=f"{basename} likely contains plaintext secrets and should be in .gitignore",
            file=rel_path, line=0,
            snippet=f"{basename} found in repo (should never be committed)",
            category="secret-storage",
        ))

    # split('\n') for parity with scan_file's readlines().
    for i, line in enumerate(text.split('\n')):
        line = core.clip_line(line)
        for rule in _RULES:
            _emit_line_findings(findings, rule, line, i + 1, rel_path)
    return findings


def main():
    args = core.parse_common_args(sys.argv, "Secret Scanner")
    repo_path = args.repo_path

    core.emit_status(args.format, f"[*] Scanning for secrets in {repo_path}...")

    ignore_patterns = core.load_ignore_patterns(repo_path)
    if ignore_patterns:
        core.emit_status(args.format, f"[*] Loaded {len(ignore_patterns)} custom ignore patterns from .forensicsignore")

    all_findings = []

    for file_path, rel_path in core.walk_repo(repo_path, ignore_patterns, skip_binary=True):
        findings = scan_file(file_path, rel_path)
        all_findings.extend(findings)

    core.output_findings(all_findings, args.format, SCANNER_NAME)


if __name__ == "__main__":
    main()
rea-analysis-optimizationmorluto/reaOptimize REA providers and shared analysis workflows for agent context, latency, CPU, and memory while preserving correctness and Evidence. Use for analysis bottlenecks and cache or query design; CI/test-suite optimization belongs to rea-ci-tests-audit.Stars: 472kUpdated Claude CodeCodexVerifiedInstall
rea-ci-tests-auditmorluto/reaAudit REA CI and test suites for correctness, fixture drift, false passes, and measured performance; use for suite or workflow investigations and efficiency changes, rather than ordinary feature testing or a single test failure.Stars: 472kUpdated Claude CodeCodexVerifiedInstall
rea-tool-designmorluto/reaAudit, investigate, design, or change REA investigation tools, CLI/MCP contracts, provider capabilities, and Evidence semantics.Stars: 472kUpdated Claude CodeCodexVerifiedInstall
ponytail-auditDietrichGebert/ponytailQuality audit of the whole repo: bugs, security, real load, missing tests, speed, and what to delete. Most important first.Stars: 194kUpdated Claude CodeCodexVerifiedInstall