Skip to content

wanshuiyin/Anti-Autoresearch / skills/experiment-forensics

experiment-forensics

Unverified

Audit experiment integrity against the evidence ledger. At L2 (repo + result files present) a fresh cross-model reviewer reads the eval code line-by-line for fake/derived ground truth, score self-normalization, phantom results (a paper number with no backing file/key), dead/uncalled metric code, verified-scope inflation, method-described ≠ method-evaluated drift, synthesized-looking results, placeholder/fake data still wired into a released result, code-output ≠ reported-number mismatch, and missing reproducibility artifacts (an empirical/agent/LLM paper shipping neither code nor the prompts/configs its results need) — every finding span-anchored to a ledger claim_id. At L0/L1 (PDF / source only) the same patterns are surfaced as info-level 'could-not-verify' signals where the ledger gives an anchor (observability_level_required:2) — NEVER a fraud verdict from a PDF. The reviewer PROPOSES findings; tools/adjudicate_findings.py computes the verdict. Detect-only. Triggers: "experiment forensics", "audit the results", "check the eval code", "实验诚实度".

Security report

Unverified

Static analysis of one exact version. It reduces risk and shows evidence; it does not certify that a skill is safe.

Commit
014a1e7cd63f1656603daaeeadb856aed1df6869
Content hash
4e2e2d13a25f753c0050002d760bcd8ce855cf8e0b3e12bf0a4793fd029b30f9
Scanned
2026-11-17
Scanner version
skillxone-scan-1.0.0
Policy version
0.1-draft
Decision
1 open medium finding to review.
Verification expires
Not applicable

Findings (1 medium, 1 info)

  • mediumPI-HIDE-FROM-USERSKILL.md:209

    The text tells the agent to keep something from the user, which hides actions taken on their behalf.

    been built and this skill never invents one — tell the user to run `/evidence-ledger`

    Suggested fix: Remove the instruction. A skill should never ask the agent to conceal what it does from the user.

    Report as false positive

  • infoINT-BROAD-ALLOWED-TOOLSSKILL.md:5

    SKILL.md pre-approves unrestricted shell access through `allowed-tools`, so the agent may run commands without asking.

    allowed-tools: Bash(*), Read, Write, Grep, Glob, mcp__codex__codex

    Suggested fix: Restrict the tool, for example `Bash(git status:*)`, or leave approval to the user.

    Report as false positive

Checks run

  • prompt-injection
  • dangerous-commands
  • credentials
  • network
  • file-access
  • integrity

Checks skipped

  • dependencies: No dependency manifest in the skill directory.
  • gitleaks: gitleaks is not installed on this host.
rea-analysis-optimizationmorluto/reaOptimize REA providers and shared analysis workflows for agent context, latency, CPU, and memory while preserving correctness and Evidence. Use for analysis bottlenecks and cache or query design; CI/test-suite optimization belongs to rea-ci-tests-audit.Stars: 472kUpdated Claude CodeCodexVerifiedInstall
rea-ci-tests-auditmorluto/reaAudit REA CI and test suites for correctness, fixture drift, false passes, and measured performance; use for suite or workflow investigations and efficiency changes, rather than ordinary feature testing or a single test failure.Stars: 472kUpdated Claude CodeCodexVerifiedInstall
rea-tool-designmorluto/reaAudit, investigate, design, or change REA investigation tools, CLI/MCP contracts, provider capabilities, and Evidence semantics.Stars: 472kUpdated Claude CodeCodexVerifiedInstall
ponytail-auditDietrichGebert/ponytailQuality audit of the whole repo: bugs, security, real load, missing tests, speed, and what to delete. Most important first.Stars: 194kUpdated Claude CodeCodexVerifiedInstall