Skip to content

gaia-research/gaia-skill-tree / .agents/skills/ev-adversarial-audit

ev-adversarial-audit

Verified

Run this skill for Phase 3 of the evidence verification pipeline — the adversarial audit step. Use when you need to check the evidence data lake for bad data before ingestion: dead links, wrong URL formats (tree/ vs blob/), subjective wording ("elite", "high-quality"), stale migration notes, benchmark catalog misuse/vendor-claim leakage, or skills whose star evidence conflicts with classified evidence level. Triggers on phrases like: "adversarial evidence check", "adversarial check", "ev-adversarial-audit", "check for noise in evidence", "flag bad evidence", "run the audit phase", "quality check the by-type files", or any reference to Phase 3 of the pipeline.

Adversarial Evidence Audit (ev-adversarial-audit)

Phase 3 audits the evidence lake from a Devil's Advocate perspective before URL health validation and ingestion.

Type-First Evidence Lake Contract (#1148)

The evidence lake is type-first. Audit evidence/by-type/<canonical-evidence-type>.md files. Legacy evidence/tier_*.md files may still exist as coexistence artifacts, but they are not the semantic routing key.

Audit Target

Split reviewer work across evidence/by-type/<type>.md files, not tier files. Suggested sharding:

  • Repo/adoption signals: repo-own, github-stars-own, social-signal
  • Technical proof: benchmark-result, arxiv, peer-review
  • Governance/proxy proof: proxy-containment, verifier-attestation
  • Composition/self proof: fusion-recipe, self-attestation

Findings to Flag

  • Dead or malformed URLs, including GitHub tree/ links where a blob/ source is required.
  • Subjective/evaluative wording not supported by the source.
  • Evidence type mismatches or legacy alias leakage.
  • Star evidence that conflicts with live verification notes from Phase 2.
  • Social/proxy/self-attestation policy violations per docs/agents/curation-authority-map.md (#2473): do not reject social-signal rows merely for being first-party (sameAsAuthor = 1.0) or published within 30 days (publishCooldownDays = 0 — no 30-day hold); instead flag social-signal rows that fail target-skill relevance (unrelated channel uploads or generic promo), lack provenance (creatorRelation / relevanceNote), fall below the 1,000-view scoring floor (views < 1000 = 0 TM), or violate (type, url) dedup / 3-row same-creator plateau. Flag any proxy-containment row pointing to a first-party, canonical, or suiteRef repository or with externalStars < 10000. Flag any synthetic self-attestation row added solely to push a suite component past the 49.9 TM shared-repository baseline cap.
  • Benchmark catalog misuse after Phase 2B: benchmark-result rows citing unknown, candidate, registered, rejected, or retired sources as if they score; vendor claims presented as reproducible benchmarks; scoring provenance on non-verified catalog entries; missing reproducibility fields or missing/dubious percentile values.
  • Stale migration notes that still treat tier_*.md as the semantic working set.
  • Multi-target peer-review packet misuse: wrong evidenceType, empty targets, invalid skillId, duplicate (source.url, skillId, evidenceType) rows, or forbidden strength/scoring fields (trustNumber, grade, class, tier, level, stars, rank).
  • Imposter over-purging & audit taxonomy conflation (RFC #1809): ensure legitimate early-stage, self-hosted, or installable: false skills (Tier 2 Packaging Gap, Tier 3 Under-Evidenced Stub) are not mislabeled as Tier 1 Malicious Imposters or flagged for deletion.

Append concise findings to the source report. Do not mutate registry files.

Optional Semantic Evidence Review Sidecar

An operator may optionally run the read-only Jev advisory runner to flag semantic anomalies in evidence rows (docs/agents/jev.md):

# Offline dry-run by default ($0 spend); add --live to query TypeSafe API
python scripts/jev_advisory.py --mode evidence --input <evidence-rows.json>

This output serves as review triage hints only. It never marks evidence rows verified, does not replace deterministic HTTP health or Phase 2B benchmark verification, does not touch Trust Magnitude scoring, and emits an actionable handoff to worker-luna on uncertainty.

rea-analysis-optimizationmorluto/reaOptimize REA providers and shared analysis workflows for agent context, latency, CPU, and memory while preserving correctness and Evidence. Use for analysis bottlenecks and cache or query design; CI/test-suite optimization belongs to rea-ci-tests-audit.Stars: 471kUpdated Claude CodeCodexVerifiedInstall
rea-ci-tests-auditmorluto/reaAudit REA CI and test suites for correctness, fixture drift, false passes, and measured performance; use for suite or workflow investigations and efficiency changes, rather than ordinary feature testing or a single test failure.Stars: 471kUpdated Claude CodeCodexVerifiedInstall
rea-tool-designmorluto/reaAudit, investigate, design, or change REA investigation tools, CLI/MCP contracts, provider capabilities, and Evidence semantics.Stars: 471kUpdated Claude CodeCodexVerifiedInstall
ponytail-auditDietrichGebert/ponytailQuality audit of the whole repo: bugs, security, real load, missing tests, speed, and what to delete. Most important first.Stars: 194kUpdated Claude CodeCodexVerifiedInstall