Skip to content

alexgreensh/repo-forensics / skills/repo-forensics

repo-forensics

Flagged

Security forensics for git repos, AI skills, and MCP servers. Audits dependencies, detects prompt injection, credential theft, runtime dynamism, manifest drift, known CVEs, CISA KEV (actively exploited) vulns, and 2026 attack patterns. Not for fixing vulnerabilities or pentesting.

FileKindSize
.gitignoreother193 B
SKILL.mdskill md43.4 KB
checksums.jsonconfig10.8 KB
checksums.json.sigother64 B
data/README.mdmarkdown1.3 KB
data/compromised_versions.jsonconfig188.3 KB
data/rule_ids.csvother35.3 KB
data/rulepacks/.gitkeepother221 B
data/rulepacks/dead_anchors.jsonconfig9.0 KB
data/rulepacks/mcp_security.jsonconfig31.2 KB
data/rulepacks/runtime_dynamism.jsonconfig46.7 KB
data/rulepacks/sast.jsonconfig107.6 KB
data/rulepacks/secrets.jsonconfig30.4 KB
data/rulepacks/shared.jsonconfig4.4 KB
data/rulepacks/skill_threats.jsonconfig95.4 KB
data/runtime_floors.jsonconfig2.3 KB
data/yara/cryptominers.yarother1.8 KB
data/yara/hacktools.yarother2.2 KB
data/yara/malware.yarother2.2 KB
data/yara/manifest.jsonconfig8.0 KB
data/yara/webshells.yarother2.5 KB
references/cursor-hook-contract.mdmarkdown5.6 KB
references/kernel_privesc_sandbox_pack.mdmarkdown5.0 KB
references/mcp-attack-patterns.mdmarkdown8.7 KB
references/research_sources.mdmarkdown8.0 KB
references/threat_patterns.mdmarkdown20.3 KB
scripts/_context_gate.pyscript21.7 KB
scripts/_ed25519.pyscript5.1 KB
scripts/_pyc_unmarshal.pyscript17.1 KB
scripts/_shared_patterns.pyscript4.8 KB
scripts/adjudication.pyscript16.4 KB
scripts/adjudication_bridge.pyscript1.0 KB
scripts/aggregate_json.pyscript69.2 KB
scripts/auto_scan.pyexecutablescript152.1 KB
scripts/corpus_sync.pyscript10.0 KB
scripts/cursor_pre_scan.pyscript22.5 KB
scripts/dast_sandbox.sbother1.6 KB
scripts/dead_anchors_extract.pyscript14.8 KB
scripts/dead_anchors_probe.pyscript27.1 KB
scripts/forensics_core.pyscript176.5 KB
scripts/gen_rule_ids.pyscript8.6 KB
scripts/hook_adapter.pyscript19.4 KB
scripts/ioc_manager.pyscript49.6 KB
scripts/parse_pnpm_lock.pyscript7.9 KB
scripts/pre_scan.pyscript143.5 KB
scripts/pypi_pins.pyscript8.1 KB
scripts/refresh_controller.pyexecutablescript48.4 KB
scripts/refresh_threat_dbs.pyexecutablescript19.4 KB
scripts/rule_loader.pyscript44.8 KB
scripts/rulepack_feed.pyscript26.4 KB
scripts/run_forensics.shexecutablescript20.3 KB
scripts/sarif_export.pyscript18.6 KB
scripts/scan_agent_skills.pyscript33.2 KB
scripts/scan_archive.pyscript29.1 KB
scripts/scan_ast.pyscript44.9 KB
scripts/scan_binary.pyscript10.9 KB
scripts/scan_bytecode.pyscript28.2 KB
scripts/scan_dast.pyscript28.8 KB
scripts/scan_dataflow.pyscript15.4 KB
scripts/scan_dead_anchors.pyscript15.9 KB
scripts/scan_decode.pyscript37.5 KB
scripts/scan_dependencies.pyscript124.5 KB
scripts/scan_devcontainer.pyscript8.5 KB
scripts/scan_entropy.pyscript15.6 KB
scripts/scan_entrypoint.pyscript34.3 KB
scripts/scan_git_config.pyscript96.3 KB
scripts/scan_git_forensics.pyscript24.1 KB
scripts/scan_history.pyscript14.5 KB
scripts/scan_infra.pyscript54.3 KB
scripts/scan_integrity.pyscript41.0 KB
scripts/scan_lifecycle.pyscript65.1 KB
scripts/scan_manifest_drift.pyscript47.0 KB
scripts/scan_mcp_security.pyscript42.3 KB
scripts/scan_oversize.pyscript11.7 KB
scripts/scan_post_incident.pyscript36.9 KB
scripts/scan_provenance.pyscript19.4 KB
scripts/scan_runtime_dynamism.pyscript23.5 KB
scripts/scan_runtime_floors.pyscript33.5 KB
scripts/scan_sast.pyscript59.1 KB
scripts/scan_secrets.pyscript11.9 KB
scripts/scan_skill_threats.pyscript102.4 KB
scripts/scan_splitstream.pyscript29.5 KB
scripts/scan_yara.pyscript19.7 KB
scripts/session_scan.pyscript54.1 KB
scripts/suppressions.pyscript15.4 KB
scripts/validate_manifests.pyscript13.7 KB
scripts/verify_install.pyscript26.2 KB
scripts/vuln_feed.pyscript31.1 KB
tests/conftest.pyscript13.4 KB
tests/corpus/DEAD_ANCHORS_CORPUS.mdmarkdown2.3 KB
tests/corpus/budgets.jsonconfig5.6 KB
tests/cursor_helpers.pyscript12.6 KB
tests/fixtures/legacy_main_blocks.jsonconfig10.3 KB
tests/fixtures/powershell_verdict_corpus.jsonconfig114.9 KB
tests/golden/parity_mcp_security.jsonconfig17.5 KB
tests/golden/parity_runtime_dynamism.jsonconfig31.7 KB
tests/golden/parity_sast.jsonconfig59.8 KB
tests/golden/parity_secrets.jsonconfig18.8 KB
tests/golden/parity_skill_threats.jsonconfig48.8 KB
tests/parity_corpus.pyscript6.3 KB
tests/powershell_hook.pyscript8.2 KB
tests/report_builders.pyscript5.0 KB
tests/schemas/sarif-schema-2.1.0.jsonconfig109.1 KB
tests/shell_compat.pyscript2.2 KB
tests/test_adjudication_bridge.pyscript832 B
tests/test_adjudication_containment.pyscript1.4 KB
tests/test_adjudication_output.pyscript20.0 KB
tests/test_adjudication_suppressions.pyscript18.7 KB
tests/test_advisory_adjudication.pyscript1.3 KB
tests/test_agent_relay_rules.pyscript7.8 KB
tests/test_aggregate_json.pyscript10.0 KB
tests/test_auto_scan_patterns.pyscript10.5 KB
tests/test_benign_corpus.pyscript16.4 KB
tests/test_benign_new_scanners.pyscript6.1 KB
tests/test_build_output_skip.pyscript10.6 KB
tests/test_capability_ledger.pyscript18.8 KB
tests/test_codex_install.pyscript10.0 KB
tests/test_codex_marketplace.pyscript3.3 KB
tests/test_comment_false_positives.pyscript2.2 KB
tests/test_composer_dependencies.pyscript4.0 KB
tests/test_compromised_versions.pyscript51.2 KB
tests/test_context_gate.pyscript32.4 KB
tests/test_coverage_status.pyscript7.6 KB
tests/test_css_compounding_shrink.pyscript20.8 KB
tests/test_cursor_adapter.pyscript7.8 KB
tests/test_cursor_benign.pyscript3.4 KB
tests/test_cursor_degrade.pyscript12.7 KB
tests/test_cursor_install.pyscript17.1 KB
tests/test_cursor_latency.pyscript7.2 KB
tests/test_cursor_parity.pyscript5.8 KB
tests/test_cursor_redteam_fn.pyscript4.9 KB
tests/test_cursor_tamper_modify.pyscript14.2 KB
tests/test_cursor_wiring.pyscript32.3 KB
tests/test_d1_yara_correlation.pyscript9.2 KB
tests/test_dead_anchors_extract.pyscript7.6 KB
tests/test_dead_anchors_probe.pyscript19.7 KB
tests/test_evidence_engine_parity.pyscript8.1 KB
tests/test_evidence_model.pyscript11.4 KB
tests/test_forensics_core.pyscript79.2 KB
tests/test_fp_hardening_r3.pyscript9.7 KB
tests/test_fp_hardening_r4.pyscript18.8 KB
tests/test_fp_regression_pairs.pyscript62.5 KB
tests/test_fp_structural.pyscript21.2 KB
tests/test_freshness.pyscript6.2 KB
tests/test_frontmatter_validity.pyscript7.0 KB
tests/test_gpg_rce_hotfix.pyscript4.8 KB
tests/test_hook_inert_data.pyscript6.1 KB
tests/test_hook_scripts.pyscript14.0 KB
tests/test_inspection_cpu_budget.pyscript11.6 KB
tests/test_ioc_manager.pyscript55.4 KB
tests/test_kernel_privesc_rules.pyscript27.8 KB
tests/test_legit_precision_regressions.pyscript6.8 KB
tests/test_lethal_trifecta.pyscript20.7 KB
tests/test_manifest_signature.pyscript1.5 KB
tests/test_manifest_validity.pyscript2.9 KB
tests/test_monotonic_trust_e2e.pyscript4.4 KB
tests/test_non_breaking_contract.pyscript22.7 KB
tests/test_openclaw_install.pyscript6.6 KB
tests/test_pack_parity.pyscript24.8 KB
tests/test_parse_pnpm_lock.pyscript10.7 KB
tests/test_plugin_checkout_provenance.pyscript11.1 KB
tests/test_powershell_hook_wire.pyscript10.6 KB
tests/test_powershell_verdict_corpus.pyscript6.2 KB
tests/test_pr44_hardening.pyscript6.5 KB
tests/test_pre_scan.pyscript19.6 KB
tests/test_pre_scan_composer.pyscript7.4 KB
tests/test_precondition_fields.pyscript4.6 KB
tests/test_provenance_offline.pyscript2.5 KB
tests/test_published_rulepack_feed_current.pyscript1.4 KB
tests/test_py38_compat.pyscript3.3 KB
tests/test_pypi_pinned_versions.pyscript19.3 KB
tests/test_pyproject_string_forms.pyscript2.2 KB
tests/test_redteam_fn_regression.pyscript32.5 KB
tests/test_reflective_fetch_execute.pyscript23.5 KB
tests/test_reflective_review_fixes.pyscript17.8 KB
tests/test_refresh_controller.pyscript19.9 KB
tests/test_refresh_rulepacks.pyscript12.4 KB
tests/test_review_gate_regressions.pyscript9.3 KB
tests/test_rule_loader.pyscript23.2 KB
tests/test_rulepack_feed.pyscript38.1 KB
tests/test_run_forensics_json.pyscript1012 B
tests/test_run_scanner_fail_loud.pyscript6.5 KB
tests/test_runtime_floors.pyscript19.8 KB
tests/test_sandbox_probing_rules.pyscript9.6 KB
tests/test_sarif_export.pyscript19.9 KB
tests/test_scan_agent_skills.pyscript31.3 KB
tests/test_scan_archive.pyscript11.6 KB
tests/test_scan_ast.pyscript12.8 KB
tests/test_scan_binary.pyscript15.3 KB
tests/test_scan_bytecode.pyscript21.5 KB
tests/test_scan_dast.pyscript32.1 KB
tests/test_scan_dataflow.pyscript7.1 KB
tests/test_scan_dead_anchors.pyscript11.5 KB
tests/test_scan_decode.pyscript28.0 KB
tests/test_scan_dependencies.pyscript37.8 KB
tests/test_scan_devcontainer.pyscript8.5 KB
tests/test_scan_entropy.pyscript11.2 KB
tests/test_scan_entrypoint.pyscript19.5 KB
tests/test_scan_git_config.pyscript58.5 KB
tests/test_scan_git_config_review_fixes.pyscript33.3 KB

scripts/scan_archive.py

View on GitHub
#!/usr/bin/env python3
"""
scan_archive.py - Archive / Document Indirection Scanner

Closes the highest-severity scanner-bypass gap from the 2026-06 CSA / Trail of
Bits audit: .zip/.docx/.jar/.whl and friends are treated as opaque binary and
never unpacked, so a payload inside a Word doc or a zip is invisible to every
scanner. (Trail of Bits used exactly this against ClawHub.)

User-safety north star (KTD6, founder directive): this scanner NEVER writes
attacker-controlled bytes to the user's disk and never spawns a subprocess per
member. Top-level archives are opened by path (read from their own on-disk
location, never re-written); nested archives are read from bounded in-memory
buffers. Members are scanned IN PROCESS. That removes the zip-slip,
symlink-escape, device-file, temp-leak, and subprocess-timeout risks an
extract-to-disk design carries.

For each archive it streams each member under a hard safety budget:
  - a running byte counter (decompressed size is MEASURED, never trusted from
    the attacker-controlled header) with per-entry, per-archive, and cumulative
    cross-nesting caps,
  - lazy member iteration so a decompression bomb (a .tgz that expands to
    gigabytes) is bounded by the entry/time caps instead of being fully
    expanded by getmembers(),
  - a compression-ratio guard that aborts a zip member as a `zip-bomb`,
  - a tar member-type allow-list: regular files only; symlink / hardlink /
    device / FIFO members are refused (a `path-traversal` finding),
  - a per-member SCAN cap and a wall-clock budget so one large member cannot
    overrun the pipeline timeout,
  - a nesting-depth cap for archives inside archives.
Surviving members are decoded and run through the shared SAST + trifecta +
secret + skill-threat detectors in process. Because the member NAME is
attacker-controlled, SAST is not gated on its extension: unknown-extension
members are sniffed as code so a payload cannot dodge detection by dropping its
extension. Inner findings are re-pathed `outer.ext -> inner/path` and the
archive is tagged `archive-indirection`.

Fail loud, never silent (R12): a budget exhaustion emits `archive-scan-incomplete`
(naming the affected archive); an archive format we cannot open
(.7z/.xz/.zst/.rar/.cab, encrypted/corrupt members) emits
`unsupported-archive-type` / `opaque-archive`; one malformed archive can never
crash the whole scan (each is isolated).

Created by Alex Greenshpun
"""

import io
import os
import re
import sys
import tarfile
import time
import zipfile
import zlib

sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
import forensics_core as core
import scan_git_config
import scan_sast
import scan_secrets
import scan_skill_threats

SCANNER_NAME = "archive"

# Zip-family (incl. OOXML office docs, jars, wheels, apks) and tar-family.
_ZIP_EXTS = {".zip", ".docx", ".xlsx", ".pptx", ".docm", ".xlsm", ".pptm",
             ".jar", ".war", ".ear", ".whl", ".egg", ".apk", ".ipa", ".nupkg"}
_TAR_EXTS = {".tar", ".tgz", ".tar.gz", ".tbz2", ".tar.bz2", ".txz", ".tar.xz"}
# Formats we deliberately do NOT support yet — surfaced loudly, not silently.
_UNSUPPORTED_EXTS = {".7z", ".xz", ".zst", ".rar", ".cab", ".lz4", ".br"}

MAX_ENTRY_BYTES = 10 * 1024 * 1024          # per-member uncompressed READ cap
MEMBER_SCAN_BYTES = 1 * 1024 * 1024         # per-member text fed to the detectors
MAX_TOTAL_UNCOMPRESSED = 100 * 1024 * 1024  # per-archive uncompressed cap
MAX_ENTRIES_PER_ARCHIVE = 1000
MAX_TOTAL_FILES = 2000                       # cumulative across all nesting levels
MAX_DEPTH = 2
RATIO_LIMIT = 100                            # uncompressed/compressed bomb threshold
TOTAL_BUDGET_SEC = 12                        # whole-scanner wall-clock (< 15s pipeline)
MAX_ARCHIVES = 2000
_CHUNK = 65536

# Exceptions a hostile/corrupt member can raise while being read. zlib.error
# (corrupt DEFLATE stream) inherits straight from Exception, NOT from any of the
# zipfile/tarfile/OS error classes, so it must be named explicitly or it crashes
# the whole scan.
_MEMBER_READ_ERRORS = (OSError, EOFError, zipfile.BadZipFile, tarfile.TarError,
                       RuntimeError, zlib.error)

# Extensions to try when a member's own extension is unrecognized — the member
# name is attacker-controlled, so a payload must not be able to dodge SAST by
# omitting or forging its extension.
_SNIFF_EXTS = (".py", ".js", ".sh")

# Leading-byte signatures. An archive renamed to a non-archive extension (e.g. a
# zip saved as `.instructions.docx.txt`, the Trail of Bits context-loader bypass)
# is invisible to extension gating, so we sniff CONTENT before deciding. tar has
# no leading magic — its `ustar` marker sits at offset 257.
_ZIP_MAGIC = (b"PK\x03\x04", b"PK\x05\x06", b"PK\x07\x08")
_UNSUPPORTED_MAGIC = (
    b"7z\xbc\xaf\x27\x1c",   # 7-Zip
    b"\xfd7zXZ\x00",         # xz
    b"Rar!\x1a\x07\x00",     # RAR4
    b"Rar!\x1a\x07\x01",     # RAR5
    b"\x28\xb5\x2f\xfd",     # zstd
    b"MSCF",                 # cab
)
_SNIFF_BYTES = 265  # enough to reach the tar `ustar` marker at offset 257

# OOXML (Office Open XML) members that are executable — a Word/Excel/PowerPoint
# document never legitimately ships a script or native binary inside its zip.
_SCRIPT_MEMBER_EXTS = (".sh", ".bash", ".zsh", ".ksh", ".py", ".pyw", ".rb",
                       ".pl", ".ps1", ".psm1", ".psd1", ".bat", ".cmd", ".command",
                       ".scpt", ".js", ".mjs", ".cjs", ".vbs", ".vbe", ".jse",
                       ".wsf", ".exe", ".dll", ".so", ".dylib", ".com", ".msi",
                       ".scr", ".app", ".deb", ".rpm", ".pkg", ".jar")


def _archive_kind(name):
    low = name.lower()
    for ext in _TAR_EXTS:  # check compound .tar.* before single exts
        if low.endswith(ext):
            return "tar"
    _, ext = os.path.splitext(low)
    if ext in _ZIP_EXTS:
        return "zip"
    if ext in _TAR_EXTS:
        return "tar"
    if ext in _UNSUPPORTED_EXTS:
        return "unsupported"
    return None


def _sniff_archive_kind(file_path):
    """Content-based archive detection by leading-byte magic. Catches an archive
    whose extension is forged or dropped. Returns the same vocabulary as
    _archive_kind ('zip'/'tar'/'unsupported') or None. gzip/bzip2 streams route
    to 'tar' because tarfile auto-detects their compression."""
    try:
        with open(file_path, "rb") as f:
            head = f.read(_SNIFF_BYTES)
    except OSError:
        return None
    if head.startswith(_ZIP_MAGIC):
        return "zip"
    if head.startswith((b"\x1f\x8b", b"BZh")):  # gzip / bzip2 stream
        return "tar"
    if len(head) >= 262 and head[257:262] == b"ustar":
        return "tar"
    for magic in _UNSUPPORTED_MAGIC:
        if head.startswith(magic):
            return "unsupported"
    # Leading magic can be hidden behind a stub (self-extracting / polyglot zip):
    # zipfile reads the end-of-central-directory at EOF, so such a file opens
    # fine yet has no PK at offset 0. is_zipfile locates the EOCD, catching it.
    try:
        if zipfile.is_zipfile(file_path):
            return "zip"
    except OSError:
        pass
    return None


def _is_ooxml(names):
    """An OOXML container has the content-types manifest plus a recognised office
    payload root. This is the structural signature, not the file extension."""
    return "[Content_Types].xml" in names and any(
        n.startswith(("word/", "xl/", "ppt/", "visio/")) for n in names)


def _is_script_member(name, data):
    """A member is a script if its name carries a script extension or its bytes
    start with a shebang. Detection is structural — payload content is irrelevant
    (a benign-looking `echo` is still an executable that should never be here)."""
    if name.lower().endswith(_SCRIPT_MEMBER_EXTS):
        return True
    return data is not None and data[:2] == b"#!"


def _office_exec_finding(label, vpath, member_name, reason):
    """A HIGH structural finding for an executable smuggled in an Office document.
    Shared by the name-based (pre-read) and shebang-based (post-read) checks."""
    return _finding(
        "high", "Executable script smuggled in Office document",
        f"{vpath} {reason} inside an OOXML document structure; Office files never "
        f"legitimately contain executable scripts.",
        label, "executable-in-office-doc", member_name)


def _finding(severity, title, desc, file, category, snippet=""):
    return core.Finding(
        scanner=SCANNER_NAME, severity=severity, title=title, description=desc,
        file=file, line=0, snippet=snippet, category=category,
    )


# Content shapes that mark a decoded member as CODE regardless of its name.
# The member name is attacker-controlled, so a zip member called README.md can
# hold a live payload — naming it after a doc previously demoted every inner
# finding to LOW and let the archive report exit 0.
_CODE_SHAPE_RE = re.compile(
    r"^\s*(?:#!|<\?php|<%|import\s+\w|from\s+[\w.]+\s+import\b|"
    r"(?:async\s+)?function\s+\w+\s*\(|def\s+\w+\s*\(|class\s+\w+\b|"
    r"(?:const|let|var)\s+\w+\s*=|package\s+\w|using\s+\w|require\s*\(|"
    r"module\.exports\b|export\s+(?:default|const|function)\b)",
    re.MULTILINE)


def _member_is_code(text):
    """True if a decoded archive member LOOKS like code/script content.

    Content-based on purpose: the evidence class of an inner finding must never
    be decided by the member or archive NAME, both of which the attacker picks.
    """
    if not text:
        return False
    return bool(core._shebang_is_code(text) or _CODE_SHAPE_RE.search(text))


def _tag_member_evidence(findings, text):
    """Pin the evidence class of a member's findings from sniffed CONTENT.

    Without this, `core.infer_evidence_class` reads the vpath — which begins
    with the ARCHIVE's name and ends with the MEMBER's name — so `notes.md`
    (a zip) containing `README.md` demoted a lethal trifecta to LOW and the
    report exited 0 with coverage=COMPLETE. A member that carries code shape is
    `direct`; anything else keeps whatever the scanner inferred."""
    if not _member_is_code(text):
        return findings
    for f in findings:
        f.evidence_class = "direct"
    return findings


# Severity ranks used when recomputing the archive wrapper's severity.
_SEV_RANK = ("critical", "high", "medium", "low")


def _effective_severity(finding):
    """A finding's severity AFTER the report layer's evidence cap.

    The wrapper summarises what the archive actually contains, so it must be
    computed from post-cap severities — otherwise it advertises a critical the
    report will grade LOW, or is itself demoted by the archive's
    doc-shaped name."""
    if getattr(finding, "evidence_class", "") in ("inferred", "structural"):
        return "low"
    return finding.severity


def _max_sev(findings):
    for sev in _SEV_RANK:
        if any(_effective_severity(f) == sev for f in findings):
            return sev
    return "low"


def _emit_inner(label, inner, findings):
    """Emit the archive-indirection summary plus the member findings, if any.
    Shared by _scan_zip and _scan_tar (identical emit block)."""
    if not inner:
        return
    # evidence_class is pinned direct: an archive containing a payload is a
    # structural FACT about bytes on disk. Inferring it from `label` let an
    # attacker name the archive `notes.md` and demote the whole wrapper.
    wrapper = _finding(_max_sev(inner), "Payload hidden inside archive",
                       f"{label} contains {len(inner)} finding(s) in its members "
                       f"(archive indirection — opaque to source scanners).",
                       label, "archive-indirection")
    wrapper.evidence_class = "direct"
    findings.append(wrapper)
    findings.extend(inner)


def _over_budget(state):
    return state["files"] >= MAX_TOTAL_FILES or (time.monotonic() - state["t0"]) > TOTAL_BUDGET_SEC


def _scan_member_text(data, vpath):
    """Run the four in-process detectors over a decoded member (bounded to
    MEMBER_SCAN_BYTES so one large member cannot overrun the budget). Returns
    inner findings carrying vpath."""
    window = data[:MEMBER_SCAN_BYTES]
    # Binary members (embedded fonts, images, compiled blobs) are not source code;
    # decoding them as text and running SAST produces garbage matches. Detect
    # binary by the RATIO of non-text bytes, NOT a single null — a lone null is
    # trivially prepended by an attacker to blind every text detector, so the
    # sentinel-byte test is unsafe. A real font/image is densely non-text; a
    # script with a stray null is overwhelmingly printable and still scanned.
    # Structural/nested-archive checks already ran in the caller.
    if window:
        nontext = sum(1 for b in window if b < 9 or (13 < b < 32))
        if nontext / len(window) > 0.30:
            return []
    text = window.decode("utf-8", errors="replace")
    out = []
    ext = os.path.splitext(vpath)[1].lower()
    if ext in scan_sast._PACK_EXTENSIONS:
        out.extend(scan_sast.scan_text(text, vpath, ext=ext))
    else:
        # Untrusted member name: don't let a missing/forged extension dodge the
        # SAST pack. Sniff as the common code languages and dedup.
        seen = set()
        for sniff in _SNIFF_EXTS:
            for f in scan_sast.scan_text(text, vpath, ext=sniff):
                key = (f.rule_id, f.line, f.title)
                if key not in seen:
                    seen.add(key)
                    out.append(f)
    out.extend(core.scan_text_trifecta(text, vpath))
    out.extend(scan_secrets.scan_text(text, vpath))
    out.extend(scan_skill_threats.scan_content(text, vpath))
    return _tag_member_evidence(out, text)


def _stream_read(fh, claimed_size, compress_size, state):
    """Read a member fileobj in chunks under the budget, MEASURING bytes (never
    trusting the header). Returns (data_or_None, status). status in
    {ok, bomb, cap, unreadable}. compress_size of 0 disables the ratio guard
    (tar members carry no per-member compressed size)."""
    if compress_size and claimed_size and (claimed_size / compress_size) > RATIO_LIMIT:
        return None, "bomb"
    chunks = []
    total = 0
    try:
        while True:
            chunk = fh.read(_CHUNK)
            if not chunk:
                break
            total += len(chunk)
            state["bytes"] += len(chunk)
            if state["bytes"] > MAX_TOTAL_UNCOMPRESSED:
                return None, "cap"
            if total > MAX_ENTRY_BYTES:
                if compress_size and (total / compress_size) > RATIO_LIMIT:
                    return None, "bomb"
                return None, "cap"
            chunks.append(chunk)
    except _MEMBER_READ_ERRORS:
        return None, "unreadable"
    return b"".join(chunks), "ok"


def _zip_source(source):
    """zipfile arg: a path (top-level, read from disk, never loaded whole) or
    bounded in-memory bytes (nested)."""
    return source if isinstance(source, str) else io.BytesIO(source)


def _scan_zip(source, label, depth, state, findings):
    try:
        zf = zipfile.ZipFile(_zip_source(source))
    except (zipfile.BadZipFile, OSError):
        findings.append(_finding("low", "Unreadable archive",
                                 f"{label} has a zip extension but is not a valid zip.",
                                 label, "opaque-archive"))
        return
    inner = []
    entries = 0
    try:
        with zf:
            try:
                infos = zf.infolist()
            except (zipfile.BadZipFile, OSError):
                findings.append(_finding("low", "Unreadable archive",
                                         f"{label} zip index is corrupt.", label, "opaque-archive"))
                return
            is_ooxml = _is_ooxml([i.filename for i in infos])
            git_roots_seen = set()
            for info in infos:
                if _over_budget(state):
                    state["incomplete"] = True
                    break
                _flag_git_member_name(
                    info.filename, label, git_roots_seen, state, inner,
                    is_dir=info.is_dir() or bool(info.external_attr & 0x10))
                if info.is_dir():
                    continue
                entries += 1
                if entries > MAX_ENTRIES_PER_ARCHIVE:
                    state["incomplete"] = True
                    break
                state["files"] += 1
                vpath = f"{label} -> {info.filename}"
                # Structural anomaly: a script inside an Office document. Flag on
                # the name alone (HIGH) so a member too large to read, or one with
                # a benign body, is still caught.
                if is_ooxml and info.filename.lower().endswith(_SCRIPT_MEMBER_EXTS):
                    inner.append(_office_exec_finding(
                        label, vpath, info.filename, "is a script member"))
                try:
                    fh = zf.open(info)
                except _MEMBER_READ_ERRORS:
                    findings.append(_finding("low", "Opaque archive member",
                                             f"Could not read {vpath} (encrypted or corrupt).",
                                             label, "opaque-archive", info.filename))
                    continue
                with fh:
                    data_m, status = _stream_read(fh, info.file_size, info.compress_size, state)
                # Shebang-based detection for script members whose extension is
                # forged (caught only once: name-based check above is extension-only).
                if (is_ooxml and status == "ok"
                        and not info.filename.lower().endswith(_SCRIPT_MEMBER_EXTS)
                        and _is_script_member(info.filename, data_m)):
                    inner.append(_office_exec_finding(
                        label, vpath, info.filename, "carries a shebang"))
                if status == "bomb":
                    findings.append(_finding("high", "Zip bomb member aborted",
                                             f"{vpath} decompresses past the safety ratio; aborted.",
                                             label, "zip-bomb", info.filename))
                    state["incomplete"] = True
                    break
                if status != "ok" or data_m is None:
                    if status == "cap":
                        state["incomplete"] = True
                    continue
                _dispatch_member(data_m, info.filename, vpath, depth, state, inner)
    finally:
        _emit_inner(label, inner, findings)


def _scan_tar(source, label, depth, state, findings):
    try:
        tf = tarfile.open(name=source) if isinstance(source, str) \
            else tarfile.open(fileobj=io.BytesIO(source))
    except (tarfile.TarError, OSError):
        findings.append(_finding("low", "Unreadable archive",
                                 f"{label} has a tar extension but is not a valid tar.",
                                 label, "opaque-archive"))
        return
    inner = []
    entries = 0
    git_roots_seen = set()
    try:
        with tf:
            # Lazy iteration: a .tgz that expands to gigabytes is bounded by the
            # entry/time caps below, NOT fully expanded by getmembers().
            tar_iter = iter(tf)
            while True:
                try:
                    member = next(tar_iter)
                except StopIteration:
                    break
                except (tarfile.TarError, OSError):
                    findings.append(_finding("low", "Unreadable archive",
                                             f"{label} tar index is corrupt.", label, "opaque-archive"))
                    break
                if _over_budget(state):
                    state["incomplete"] = True
                    break
                _flag_git_member_name(member.name, label, git_roots_seen,
                                      state, inner, is_dir=member.isdir())
                if not member.isfile():
                    if member.issym() or member.islnk():
                        findings.append(_finding("high", "Unsafe tar link member refused",
                                                 f"{label} contains a {'sym' if member.issym() else 'hard'}link "
                                                 f"member ({member.name}) that can point outside the archive; refused.",
                                                 label, "path-traversal", member.name))
                    elif member.ischr() or member.isblk() or member.isfifo():
                        findings.append(_finding("high", "Unsafe tar device member refused",
                                                 f"{label} contains a device/FIFO member ({member.name}); refused.",
                                                 label, "path-traversal", member.name))
                    continue
                entries += 1
                if entries > MAX_ENTRIES_PER_ARCHIVE:
                    state["incomplete"] = True
                    break
                state["files"] += 1
                vpath = f"{label} -> {member.name}"
                try:
                    fh = tf.extractfile(member)
                except (tarfile.TarError, OSError):
                    fh = None
                if fh is None:
                    continue
                with fh:
                    # compress_size=0: tar members carry no per-member compressed
                    # size, so the ratio guard is disabled and absolute caps apply.
                    data_m, status = _stream_read(fh, member.size, 0, state)
                if status != "ok" or data_m is None:
                    if status == "cap":
                        state["incomplete"] = True
                    continue
                _dispatch_member(data_m, member.name, vpath, depth, state, inner)
    finally:
        _emit_inner(label, inner, findings)


def _flag_git_member_name(member_name, label, git_roots_seen, state, inner,
                          is_dir=False):
    """Name-only shipped-.git detection for archive members. Runs before any
    read so an unreadable, oversized, or directory member is still surfaced:
    an archive that carries git metadata is hostile by construction. is_dir
    carries the archive's own directory marking (zip DOS dir bit / trailing
    slash, tar dir member) so a bare `.git` directory entry is shipped
    metadata (critical), not a gitdir pointer file (high)."""
    cls = scan_git_config.classify_git_member_path(member_name, is_dir=is_dir)
    if not cls:
        return
    kind, git_root = cls
    if kind == "gitdir_file":
        inner.append(scan_git_config.archive_gitdir_finding(label, member_name))
        return
    if git_root not in git_roots_seen:
        git_roots_seen.add(git_root)
        inner.append(scan_git_config.archive_shipped_git_finding(label, git_root))
    if kind == "hooks" and state.get("git_hooks", 0) < scan_git_config._MAX_HOOK_FINDINGS:
        state["git_hooks"] = state.get("git_hooks", 0) + 1
        inner.append(scan_git_config.archive_shipped_hooks_finding(label, member_name))


def _dispatch_member(data, member_name, vpath, depth, state, inner):
    """Scan a member; recurse into nested archives up to MAX_DEPTH."""
    kind = _archive_kind(member_name)
    if kind in ("zip", "tar"):
        if depth >= MAX_DEPTH:
            inner.append(_finding("low", "Nested archive past depth limit",
                                  f"{vpath} is a nested archive beyond depth {MAX_DEPTH}; not opened.",
                                  vpath, "archive-scan-incomplete"))
            return
        nested = []
        if kind == "zip":
            _scan_zip(data, vpath, depth + 1, state, nested)
        else:
            _scan_tar(data, vpath, depth + 1, state, nested)
        inner.extend(nested)
        return
    if kind == "unsupported":
        inner.append(_finding("low", "Unsupported nested archive type",
                              f"{vpath} is an archive format this scanner cannot open.",
                              vpath, "unsupported-archive-type"))
        return
    inner.extend(_scan_member_text(data, vpath))
    inner.extend(scan_git_config.scan_archive_member_content(data, member_name, vpath))


def scan_repo(repo_path, ignore_patterns=None):
    all_findings = []
    state = {"files": 0, "bytes": 0, "t0": time.monotonic(), "incomplete": False}
    starved = []  # archives that tripped a cap, in visit order (T5: name them)
    archives = 0
    for file_path, rel_path in core.walk_aux(
        repo_path, ignore_patterns=ignore_patterns, apply_size_cap=False
    ):
        kind = _archive_kind(os.path.basename(file_path))
        if kind is None:
            # Extension said "not an archive" — sniff CONTENT before trusting it.
            # An archive renamed to dodge extension gating is caught here.
            kind = _sniff_archive_kind(file_path)
        if kind is None:
            continue
        if archives >= MAX_ARCHIVES:
            all_findings.append(_finding("medium", "Archive scan incomplete (archive-count budget)",
                                         "Reached the maximum number of archives to scan; "
                                         "remaining archives not inspected.",
                                         rel_path, "archive-scan-incomplete"))
            break
        archives += 1

        if kind == "unsupported":
            all_findings.append(_finding("low", "Unsupported archive type",
                                         f"{rel_path} is an archive format this scanner cannot open "
                                         f"(.7z/.xz/.zst/.rar/.cab). Contents not inspected.",
                                         rel_path, "unsupported-archive-type"))
            continue

        # Fail loud per-archive (R12): if the cumulative budget is already spent,
        # name THIS archive as uninspected rather than emitting one generic note
        # that hides which archive a fan-out decoy starved out.
        if _over_budget(state):
            all_findings.append(_finding("medium", "Archive not scanned (budget exhausted)",
                                         f"{rel_path} was reached after the cumulative file/time budget "
                                         f"was spent (often by an earlier fan-out archive); not inspected.",
                                         rel_path, "archive-scan-incomplete"))
            continue

        # Reset per-archive byte budget (cumulative file count persists). Each
        # archive is isolated: a malformed one can never crash the whole scan.
        state["bytes"] = 0
        # Snapshot the shared flag so a cap tripped while scanning THIS archive
        # can be attributed to it. state["incomplete"] is global and sticky, and
        # the caps are checked deep inside the member loops, so this edge-detect
        # at the call site is what turns "something, somewhere was truncated"
        # into a name (T5).
        _was_incomplete = state["incomplete"]
        try:
            if kind == "zip":
                _scan_zip(file_path, rel_path, 0, state, all_findings)
            else:
                _scan_tar(file_path, rel_path, 0, state, all_findings)
        except Exception as exc:  # noqa: BLE001 - last-resort isolation per archive
            all_findings.append(_finding("low", "Archive could not be scanned",
                                         f"{rel_path} raised an unexpected error during scan "
                                         f"({type(exc).__name__}); skipped.",
                                         rel_path, "opaque-archive"))
        if state["incomplete"] and not _was_incomplete:
            starved.append(rel_path)

    if state["incomplete"]:
        # Name the archives that were actually truncated. "One or more archives
        # hit a safety cap" told an operator that something was missed but not
        # WHERE, which is indistinguishable from a silent skip for anyone trying
        # to act on it -- the archive that got starved is exactly the one worth
        # re-scanning on its own. Falls back to the repo path only when the
        # truncation could not be attributed to a specific archive.
        if starved:
            named = ", ".join(starved[:10])
            more = f" (+{len(starved) - 10} more)" if len(starved) > 10 else ""
            all_findings.append(_finding("low", "Archive scan incomplete",
                                         f"Hit a safety cap (entry count, size, ratio, time, or "
                                         f"depth) while scanning: {named}{more}. Some members were "
                                         f"not fully inspected; re-scan these archives directly.",
                                         starved[0], "archive-scan-incomplete"))
        else:
            all_findings.append(_finding("low", "Archive scan incomplete",
                                         "One or more archives hit a safety cap (entry count, size, "
                                         "ratio, time, or depth); some members were not fully inspected.",
                                         repo_path, "archive-scan-incomplete"))
    return all_findings


def main():
    args = core.parse_common_args(sys.argv, "Archive Indirection Scanner")
    repo_path = args.repo_path
    core.emit_status(args.format, f"[*] Scanning archives in {repo_path}...")
    ignore_patterns = core.load_ignore_patterns(repo_path)
    all_findings = scan_repo(repo_path, ignore_patterns)
    core.output_findings(all_findings, args.format, SCANNER_NAME)


if __name__ == "__main__":
    main()
rea-analysis-optimizationmorluto/reaOptimize REA providers and shared analysis workflows for agent context, latency, CPU, and memory while preserving correctness and Evidence. Use for analysis bottlenecks and cache or query design; CI/test-suite optimization belongs to rea-ci-tests-audit.Stars: 472kUpdated Claude CodeCodexVerifiedInstall
rea-ci-tests-auditmorluto/reaAudit REA CI and test suites for correctness, fixture drift, false passes, and measured performance; use for suite or workflow investigations and efficiency changes, rather than ordinary feature testing or a single test failure.Stars: 472kUpdated Claude CodeCodexVerifiedInstall
rea-tool-designmorluto/reaAudit, investigate, design, or change REA investigation tools, CLI/MCP contracts, provider capabilities, and Evidence semantics.Stars: 472kUpdated Claude CodeCodexVerifiedInstall
ponytail-auditDietrichGebert/ponytailQuality audit of the whole repo: bugs, security, real load, missing tests, speed, and what to delete. Most important first.Stars: 194kUpdated Claude CodeCodexVerifiedInstall