Skip to content

alexgreensh/repo-forensics / skills/repo-forensics

repo-forensics

Flagged

Security forensics for git repos, AI skills, and MCP servers. Audits dependencies, detects prompt injection, credential theft, runtime dynamism, manifest drift, known CVEs, CISA KEV (actively exploited) vulns, and 2026 attack patterns. Not for fixing vulnerabilities or pentesting.

FileKindSize
.gitignoreother193 B
SKILL.mdskill md43.4 KB
checksums.jsonconfig10.8 KB
checksums.json.sigother64 B
data/README.mdmarkdown1.3 KB
data/compromised_versions.jsonconfig188.3 KB
data/rule_ids.csvother35.3 KB
data/rulepacks/.gitkeepother221 B
data/rulepacks/dead_anchors.jsonconfig9.0 KB
data/rulepacks/mcp_security.jsonconfig31.2 KB
data/rulepacks/runtime_dynamism.jsonconfig46.7 KB
data/rulepacks/sast.jsonconfig107.6 KB
data/rulepacks/secrets.jsonconfig30.4 KB
data/rulepacks/shared.jsonconfig4.4 KB
data/rulepacks/skill_threats.jsonconfig95.4 KB
data/runtime_floors.jsonconfig2.3 KB
data/yara/cryptominers.yarother1.8 KB
data/yara/hacktools.yarother2.2 KB
data/yara/malware.yarother2.2 KB
data/yara/manifest.jsonconfig8.0 KB
data/yara/webshells.yarother2.5 KB
references/cursor-hook-contract.mdmarkdown5.6 KB
references/kernel_privesc_sandbox_pack.mdmarkdown5.0 KB
references/mcp-attack-patterns.mdmarkdown8.7 KB
references/research_sources.mdmarkdown8.0 KB
references/threat_patterns.mdmarkdown20.3 KB
scripts/_context_gate.pyscript21.7 KB
scripts/_ed25519.pyscript5.1 KB
scripts/_pyc_unmarshal.pyscript17.1 KB
scripts/_shared_patterns.pyscript4.8 KB
scripts/adjudication.pyscript16.4 KB
scripts/adjudication_bridge.pyscript1.0 KB
scripts/aggregate_json.pyscript69.2 KB
scripts/auto_scan.pyexecutablescript152.1 KB
scripts/corpus_sync.pyscript10.0 KB
scripts/cursor_pre_scan.pyscript22.5 KB
scripts/dast_sandbox.sbother1.6 KB
scripts/dead_anchors_extract.pyscript14.8 KB
scripts/dead_anchors_probe.pyscript27.1 KB
scripts/forensics_core.pyscript176.5 KB
scripts/gen_rule_ids.pyscript8.6 KB
scripts/hook_adapter.pyscript19.4 KB
scripts/ioc_manager.pyscript49.6 KB
scripts/parse_pnpm_lock.pyscript7.9 KB
scripts/pre_scan.pyscript143.5 KB
scripts/pypi_pins.pyscript8.1 KB
scripts/refresh_controller.pyexecutablescript48.4 KB
scripts/refresh_threat_dbs.pyexecutablescript19.4 KB
scripts/rule_loader.pyscript44.8 KB
scripts/rulepack_feed.pyscript26.4 KB
scripts/run_forensics.shexecutablescript20.3 KB
scripts/sarif_export.pyscript18.6 KB
scripts/scan_agent_skills.pyscript33.2 KB
scripts/scan_archive.pyscript29.1 KB
scripts/scan_ast.pyscript44.9 KB
scripts/scan_binary.pyscript10.9 KB
scripts/scan_bytecode.pyscript28.2 KB
scripts/scan_dast.pyscript28.8 KB
scripts/scan_dataflow.pyscript15.4 KB
scripts/scan_dead_anchors.pyscript15.9 KB
scripts/scan_decode.pyscript37.5 KB
scripts/scan_dependencies.pyscript124.5 KB
scripts/scan_devcontainer.pyscript8.5 KB
scripts/scan_entropy.pyscript15.6 KB
scripts/scan_entrypoint.pyscript34.3 KB
scripts/scan_git_config.pyscript96.3 KB
scripts/scan_git_forensics.pyscript24.1 KB
scripts/scan_history.pyscript14.5 KB
scripts/scan_infra.pyscript54.3 KB
scripts/scan_integrity.pyscript41.0 KB
scripts/scan_lifecycle.pyscript65.1 KB
scripts/scan_manifest_drift.pyscript47.0 KB
scripts/scan_mcp_security.pyscript42.3 KB
scripts/scan_oversize.pyscript11.7 KB
scripts/scan_post_incident.pyscript36.9 KB
scripts/scan_provenance.pyscript19.4 KB
scripts/scan_runtime_dynamism.pyscript23.5 KB
scripts/scan_runtime_floors.pyscript33.5 KB
scripts/scan_sast.pyscript59.1 KB
scripts/scan_secrets.pyscript11.9 KB
scripts/scan_skill_threats.pyscript102.4 KB
scripts/scan_splitstream.pyscript29.5 KB
scripts/scan_yara.pyscript19.7 KB
scripts/session_scan.pyscript54.1 KB
scripts/suppressions.pyscript15.4 KB
scripts/validate_manifests.pyscript13.7 KB
scripts/verify_install.pyscript26.2 KB
scripts/vuln_feed.pyscript31.1 KB
tests/conftest.pyscript13.4 KB
tests/corpus/DEAD_ANCHORS_CORPUS.mdmarkdown2.3 KB
tests/corpus/budgets.jsonconfig5.6 KB
tests/cursor_helpers.pyscript12.6 KB
tests/fixtures/legacy_main_blocks.jsonconfig10.3 KB
tests/fixtures/powershell_verdict_corpus.jsonconfig114.9 KB
tests/golden/parity_mcp_security.jsonconfig17.5 KB
tests/golden/parity_runtime_dynamism.jsonconfig31.7 KB
tests/golden/parity_sast.jsonconfig59.8 KB
tests/golden/parity_secrets.jsonconfig18.8 KB
tests/golden/parity_skill_threats.jsonconfig48.8 KB
tests/parity_corpus.pyscript6.3 KB
tests/powershell_hook.pyscript8.2 KB
tests/report_builders.pyscript5.0 KB
tests/schemas/sarif-schema-2.1.0.jsonconfig109.1 KB
tests/shell_compat.pyscript2.2 KB
tests/test_adjudication_bridge.pyscript832 B
tests/test_adjudication_containment.pyscript1.4 KB
tests/test_adjudication_output.pyscript20.0 KB
tests/test_adjudication_suppressions.pyscript18.7 KB
tests/test_advisory_adjudication.pyscript1.3 KB
tests/test_agent_relay_rules.pyscript7.8 KB
tests/test_aggregate_json.pyscript10.0 KB
tests/test_auto_scan_patterns.pyscript10.5 KB
tests/test_benign_corpus.pyscript16.4 KB
tests/test_benign_new_scanners.pyscript6.1 KB
tests/test_build_output_skip.pyscript10.6 KB
tests/test_capability_ledger.pyscript18.8 KB
tests/test_codex_install.pyscript10.0 KB
tests/test_codex_marketplace.pyscript3.3 KB
tests/test_comment_false_positives.pyscript2.2 KB
tests/test_composer_dependencies.pyscript4.0 KB
tests/test_compromised_versions.pyscript51.2 KB
tests/test_context_gate.pyscript32.4 KB
tests/test_coverage_status.pyscript7.6 KB
tests/test_css_compounding_shrink.pyscript20.8 KB
tests/test_cursor_adapter.pyscript7.8 KB
tests/test_cursor_benign.pyscript3.4 KB
tests/test_cursor_degrade.pyscript12.7 KB
tests/test_cursor_install.pyscript17.1 KB
tests/test_cursor_latency.pyscript7.2 KB
tests/test_cursor_parity.pyscript5.8 KB
tests/test_cursor_redteam_fn.pyscript4.9 KB
tests/test_cursor_tamper_modify.pyscript14.2 KB
tests/test_cursor_wiring.pyscript32.3 KB
tests/test_d1_yara_correlation.pyscript9.2 KB
tests/test_dead_anchors_extract.pyscript7.6 KB
tests/test_dead_anchors_probe.pyscript19.7 KB
tests/test_evidence_engine_parity.pyscript8.1 KB
tests/test_evidence_model.pyscript11.4 KB
tests/test_forensics_core.pyscript79.2 KB
tests/test_fp_hardening_r3.pyscript9.7 KB
tests/test_fp_hardening_r4.pyscript18.8 KB
tests/test_fp_regression_pairs.pyscript62.5 KB
tests/test_fp_structural.pyscript21.2 KB
tests/test_freshness.pyscript6.2 KB
tests/test_frontmatter_validity.pyscript7.0 KB
tests/test_gpg_rce_hotfix.pyscript4.8 KB
tests/test_hook_inert_data.pyscript6.1 KB
tests/test_hook_scripts.pyscript14.0 KB
tests/test_inspection_cpu_budget.pyscript11.6 KB
tests/test_ioc_manager.pyscript55.4 KB
tests/test_kernel_privesc_rules.pyscript27.8 KB
tests/test_legit_precision_regressions.pyscript6.8 KB
tests/test_lethal_trifecta.pyscript20.7 KB
tests/test_manifest_signature.pyscript1.5 KB
tests/test_manifest_validity.pyscript2.9 KB
tests/test_monotonic_trust_e2e.pyscript4.4 KB
tests/test_non_breaking_contract.pyscript22.7 KB
tests/test_openclaw_install.pyscript6.6 KB
tests/test_pack_parity.pyscript24.8 KB
tests/test_parse_pnpm_lock.pyscript10.7 KB
tests/test_plugin_checkout_provenance.pyscript11.1 KB
tests/test_powershell_hook_wire.pyscript10.6 KB
tests/test_powershell_verdict_corpus.pyscript6.2 KB
tests/test_pr44_hardening.pyscript6.5 KB
tests/test_pre_scan.pyscript19.6 KB
tests/test_pre_scan_composer.pyscript7.4 KB
tests/test_precondition_fields.pyscript4.6 KB
tests/test_provenance_offline.pyscript2.5 KB
tests/test_published_rulepack_feed_current.pyscript1.4 KB
tests/test_py38_compat.pyscript3.3 KB
tests/test_pypi_pinned_versions.pyscript19.3 KB
tests/test_pyproject_string_forms.pyscript2.2 KB
tests/test_redteam_fn_regression.pyscript32.5 KB
tests/test_reflective_fetch_execute.pyscript23.5 KB
tests/test_reflective_review_fixes.pyscript17.8 KB
tests/test_refresh_controller.pyscript19.9 KB
tests/test_refresh_rulepacks.pyscript12.4 KB
tests/test_review_gate_regressions.pyscript9.3 KB
tests/test_rule_loader.pyscript23.2 KB
tests/test_rulepack_feed.pyscript38.1 KB
tests/test_run_forensics_json.pyscript1012 B
tests/test_run_scanner_fail_loud.pyscript6.5 KB
tests/test_runtime_floors.pyscript19.8 KB
tests/test_sandbox_probing_rules.pyscript9.6 KB
tests/test_sarif_export.pyscript19.9 KB
tests/test_scan_agent_skills.pyscript31.3 KB
tests/test_scan_archive.pyscript11.6 KB
tests/test_scan_ast.pyscript12.8 KB
tests/test_scan_binary.pyscript15.3 KB
tests/test_scan_bytecode.pyscript21.5 KB
tests/test_scan_dast.pyscript32.1 KB
tests/test_scan_dataflow.pyscript7.1 KB
tests/test_scan_dead_anchors.pyscript11.5 KB
tests/test_scan_decode.pyscript28.0 KB
tests/test_scan_dependencies.pyscript37.8 KB
tests/test_scan_devcontainer.pyscript8.5 KB
tests/test_scan_entropy.pyscript11.2 KB
tests/test_scan_entrypoint.pyscript19.5 KB
tests/test_scan_git_config.pyscript58.5 KB
tests/test_scan_git_config_review_fixes.pyscript33.3 KB

scripts/corpus_sync.py

View on GitHub
#!/usr/bin/env python3
"""
corpus_sync.py - Dev-only tool: snapshots pinned-SHA repositories into
~/.cache/repo-forensics/corpus/ for use with the extended-corpus regression
gate in tests/test_benign_corpus.py.

THIS SCRIPT IS THE ONLY NETWORK-TOUCHING PIECE OF THE BENIGN CORPUS SYSTEM.
It is intended for explicit developer invocation only and is NEVER imported
by the test suite.  Calling it from a test is a bug.

Usage:
    python3 scripts/corpus_sync.py [--cache-dir PATH] [--dry-run] [--verbose]

The script downloads the HEAD of each pinned repository SHA from GitHub via
HTTPS (host allowlist: github.com, raw.githubusercontent.com only), extracts
files matching benign file patterns, and writes them to:

    ~/.cache/repo-forensics/corpus/<repo-slug>/<sha-short>/

Re-runs are idempotent: a directory whose sha-short already exists is skipped.

Network discipline (mirrors ioc_manager.py):
- HTTPS only; http:// URLs are rejected before any connection.
- Allowlisted hosts: ALLOWED_HOSTS below; any other host is rejected.
- 5 MB per-file cap; zip archives capped at 20 MB.
- Atomic temp+rename writes; cache dir mode 0700.
- No credentials stored or transmitted.
- Content-type is validated; JSON-decoded archives only.

Run tests/test_benign_corpus.py without this script: the extended-corpus
parameterised tests skip gracefully when the cache dir is absent.

Created by Alex Greenshpun (U7, 2026-06-10)
"""

import argparse
import io
import json
import os
import pathlib
import sys
import tempfile
import time
import urllib.error
import urllib.request
import zipfile

# ---------------------------------------------------------------------------
# Network discipline
# ---------------------------------------------------------------------------

ALLOWED_HOSTS = frozenset({
    "github.com",
    "raw.githubusercontent.com",
    "codeload.github.com",
})

MAX_ARCHIVE_BYTES = 20 * 1024 * 1024   # 20 MB
MAX_FILE_BYTES = 5 * 1024 * 1024        # 5 MB
REQUEST_TIMEOUT_SECONDS = 30


def _safe_fetch(url, max_bytes=MAX_ARCHIVE_BYTES):
    """Fetch *url* with host-allowlist and size-cap enforcement.

    Returns bytes on success.  Raises ValueError for policy violations;
    raises urllib.error.URLError / IOError for network failures.
    """
    if not url.startswith("https://"):
        raise ValueError(f"Non-HTTPS URL rejected: {url!r}")
    from urllib.parse import urlparse
    host = urlparse(url).hostname or ""
    if host not in ALLOWED_HOSTS:
        raise ValueError(f"Host not in allowlist: {host!r}")

    req = urllib.request.Request(
        url,
        headers={"User-Agent": "repo-forensics-corpus-sync/1.0"},
    )
    with urllib.request.urlopen(req, timeout=REQUEST_TIMEOUT_SECONDS) as resp:
        content_length = resp.headers.get("Content-Length")
        if content_length and int(content_length) > max_bytes:
            raise ValueError(
                f"Content-Length {content_length} exceeds cap {max_bytes}"
            )
        data = resp.read(max_bytes + 1)
        if len(data) > max_bytes:
            raise ValueError(
                f"Response body exceeds cap {max_bytes} bytes"
            )
    return data


# ---------------------------------------------------------------------------
# Pinned corpus entries
# ---------------------------------------------------------------------------
# Each entry specifies a publicly-accessible GitHub repository at an exact
# commit SHA.  Only files whose names or extensions match BENIGN_PATTERNS are
# extracted -- never compiled code, archives, or binaries.
#
# Adding a new entry: choose a well-known open-source project with a
# permissive license; pick a recent release tag SHA; verify locally that the
# extracted files produce zero critical/high findings before committing here.

PINNED_REPOS = [
    {
        "slug": "github_linguist",
        "repo": "github-linguist/linguist",
        "sha": "a7b8a0b1b77fc7f3d62c1ed1e0e53e19fdb08a3a",  # v7.29.0
        "description": "GitHub Linguist -- large benign codebase, many file types",
        "extract_globs": ["*.md", "*.txt", "*.json", "*.yml", "*.yaml"],
        "max_files": 50,
    },
    {
        "slug": "actions_checkout",
        "repo": "actions/checkout",
        "sha": "11bd71901bbe5b1630ceea73d27597364c9af683",  # v4.2.2
        "description": "actions/checkout -- clean GitHub Actions workflow and JS",
        "extract_globs": ["*.yml", "*.json", "*.md"],
        "max_files": 30,
    },
]

# File extensions/names accepted from any pinned repo.
BENIGN_EXTENSIONS = frozenset({
    ".md", ".txt", ".rst", ".yml", ".yaml", ".json", ".toml",
    ".cfg", ".ini", ".css", ".html",
})
BENIGN_BASENAMES = frozenset({
    "Dockerfile", ".env.example", ".env.template",
    "Makefile", "LICENSE", "NOTICE",
})


def _should_extract(name):
    """Return True if the file's basename / extension is in the benign set."""
    base = os.path.basename(name)
    ext = os.path.splitext(base)[1].lower()
    return ext in BENIGN_EXTENSIONS or base in BENIGN_BASENAMES


# ---------------------------------------------------------------------------
# Sync logic
# ---------------------------------------------------------------------------

def _sha_short(sha):
    return sha[:12]


def _cache_dir_for(base_cache, entry):
    return base_cache / entry["slug"] / _sha_short(entry["sha"])


def sync_entry(entry, base_cache, dry_run=False, verbose=False):
    """Download and extract files for one pinned repo entry."""
    target_dir = _cache_dir_for(base_cache, entry)

    if target_dir.exists():
        if verbose:
            print(f"[skip] {entry['slug']} @ {_sha_short(entry['sha'])} already cached")
        return

    repo = entry["repo"]
    sha = entry["sha"]
    url = f"https://codeload.github.com/{repo}/zip/{sha}"

    if verbose:
        print(f"[fetch] {entry['slug']} @ {_sha_short(sha)} from {url}")

    if dry_run:
        print(f"[dry-run] would fetch: {url}")
        return

    # Fetch archive
    try:
        archive_bytes = _safe_fetch(url, max_bytes=MAX_ARCHIVE_BYTES)
    except (urllib.error.URLError, ValueError, OSError) as exc:
        print(f"[error] fetch failed for {entry['slug']}: {exc}", file=sys.stderr)
        return

    # Validate zip structure
    try:
        zf = zipfile.ZipFile(io.BytesIO(archive_bytes))
    except zipfile.BadZipFile as exc:
        print(f"[error] invalid zip for {entry['slug']}: {exc}", file=sys.stderr)
        return

    # Extract to temp dir first (atomic rename on success)
    tmp_parent = base_cache / entry["slug"]
    tmp_parent.mkdir(parents=True, exist_ok=True)
    tmp_parent.chmod(0o700)

    with tempfile.TemporaryDirectory(dir=str(tmp_parent)) as tmp_dir:
        tmp_path = pathlib.Path(tmp_dir) / _sha_short(sha)
        tmp_path.mkdir()

        max_files = entry.get("max_files", 100)
        extracted = 0
        for info in zf.infolist():
            if info.is_dir():
                continue
            if not _should_extract(info.filename):
                continue
            if info.file_size > MAX_FILE_BYTES:
                if verbose:
                    print(f"  [skip-large] {info.filename} ({info.file_size} bytes)")
                continue
            if extracted >= max_files:
                if verbose:
                    print(f"  [cap] reached max_files={max_files}, stopping extraction")
                break

            # Flatten path: keep only basename to avoid path-traversal risk.
            safe_name = os.path.basename(info.filename)
            if not safe_name:
                continue

            dest = tmp_path / safe_name
            # Resolve and confirm it's inside tmp_path (path traversal guard)
            try:
                dest.resolve().relative_to(tmp_path.resolve())
            except ValueError:
                print(f"  [path-traversal] rejected: {info.filename}", file=sys.stderr)
                continue

            dest.write_bytes(zf.read(info.filename))
            extracted += 1
            if verbose:
                print(f"  [extract] {info.filename} -> {safe_name}")

        # Atomic rename
        target_dir.parent.mkdir(parents=True, exist_ok=True)
        tmp_path.rename(target_dir)
        target_dir.chmod(0o700)

    if verbose:
        print(f"[done] {entry['slug']} @ {_sha_short(sha)}: {extracted} files -> {target_dir}")


def sync_all(base_cache, dry_run=False, verbose=False):
    """Sync all pinned entries into base_cache."""
    base_cache = pathlib.Path(base_cache)
    base_cache.mkdir(parents=True, exist_ok=True)
    base_cache.chmod(0o700)

    # Write a metadata file for idempotency and audit.
    meta_path = base_cache / "corpus_sync_meta.json"
    meta = {
        "synced_at": time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime()),
        "entries": [
            {"slug": e["slug"], "sha": e["sha"], "repo": e["repo"]}
            for e in PINNED_REPOS
        ],
    }

    for entry in PINNED_REPOS:
        sync_entry(entry, base_cache, dry_run=dry_run, verbose=verbose)

    if not dry_run:
        with open(meta_path, "w", encoding="utf-8") as fh:
            json.dump(meta, fh, indent=2)

    print(f"[corpus_sync] done. Cache: {base_cache}")


# ---------------------------------------------------------------------------
# CLI
# ---------------------------------------------------------------------------

def main():
    parser = argparse.ArgumentParser(
        description=__doc__,
        formatter_class=argparse.RawDescriptionHelpFormatter,
    )
    parser.add_argument(
        "--cache-dir",
        default=str(
            pathlib.Path(os.path.expanduser("~")) / ".cache" / "repo-forensics" / "corpus"
        ),
        help="Override cache directory (default: ~/.cache/repo-forensics/corpus/)",
    )
    parser.add_argument(
        "--dry-run",
        action="store_true",
        help="Print what would be fetched without fetching",
    )
    parser.add_argument(
        "--verbose",
        action="store_true",
        help="Print per-file extraction progress",
    )
    args = parser.parse_args()
    sync_all(args.cache_dir, dry_run=args.dry_run, verbose=args.verbose)


if __name__ == "__main__":
    main()
rea-analysis-optimizationmorluto/reaOptimize REA providers and shared analysis workflows for agent context, latency, CPU, and memory while preserving correctness and Evidence. Use for analysis bottlenecks and cache or query design; CI/test-suite optimization belongs to rea-ci-tests-audit.Stars: 472kUpdated Claude CodeCodexVerifiedInstall
rea-ci-tests-auditmorluto/reaAudit REA CI and test suites for correctness, fixture drift, false passes, and measured performance; use for suite or workflow investigations and efficiency changes, rather than ordinary feature testing or a single test failure.Stars: 472kUpdated Claude CodeCodexVerifiedInstall
rea-tool-designmorluto/reaAudit, investigate, design, or change REA investigation tools, CLI/MCP contracts, provider capabilities, and Evidence semantics.Stars: 472kUpdated Claude CodeCodexVerifiedInstall
ponytail-auditDietrichGebert/ponytailQuality audit of the whole repo: bugs, security, real load, missing tests, speed, and what to delete. Most important first.Stars: 194kUpdated Claude CodeCodexVerifiedInstall