Skip to content

gaia-research/gaia-skill-tree / .agents/skills/ev-adversarial-audit

ev-adversarial-audit

Verified

Run this skill for Phase 3 of the evidence verification pipeline — the adversarial audit step. Use when you need to check the evidence data lake for bad data before ingestion: dead links, wrong URL formats (tree/ vs blob/), subjective wording ("elite", "high-quality"), stale migration notes, benchmark catalog misuse/vendor-claim leakage, or skills whose star evidence conflicts with classified evidence level. Triggers on phrases like: "adversarial evidence check", "adversarial check", "ev-adversarial-audit", "check for noise in evidence", "flag bad evidence", "run the audit phase", "quality check the by-type files", or any reference to Phase 3 of the pipeline.

FileKindSize
SKILL.mdskill md4.3 KB
---
name: ev-adversarial-audit
description: >
  Run this skill for Phase 3 of the evidence verification pipeline — the adversarial audit step. Use when you need to check the evidence data lake for bad data before ingestion: dead links, wrong URL formats (tree/ vs blob/), subjective wording ("elite", "high-quality"), stale migration notes, benchmark catalog misuse/vendor-claim leakage, or skills whose star evidence conflicts with classified evidence level. Triggers on phrases like: "adversarial evidence check", "adversarial check", "ev-adversarial-audit", "check for noise in evidence", "flag bad evidence", "run the audit phase", "quality check the by-type files", or any reference to Phase 3 of the pipeline.
---

# Adversarial Evidence Audit (ev-adversarial-audit)

Phase 3 audits the evidence lake from a Devil's Advocate perspective before URL health validation and ingestion.

## Type-First Evidence Lake Contract (#1148)

The evidence lake is **type-first**. Audit `evidence/by-type/<canonical-evidence-type>.md` files. Legacy `evidence/tier_*.md` files may still exist as coexistence artifacts, but they are **not** the semantic routing key.

## Audit Target

Split reviewer work across `evidence/by-type/<type>.md` files, not tier files. Suggested sharding:

- Repo/adoption signals: `repo-own`, `github-stars-own`, `social-signal`
- Technical proof: `benchmark-result`, `arxiv`, `peer-review`
- Governance/proxy proof: `proxy-containment`, `verifier-attestation`
- Composition/self proof: `fusion-recipe`, `self-attestation`

## Findings to Flag

- Dead or malformed URLs, including GitHub `tree/` links where a `blob/` source is required.
- Subjective/evaluative wording not supported by the source.
- Evidence type mismatches or legacy alias leakage.
- Star evidence that conflicts with live verification notes from Phase 2.
- Social/proxy/self-attestation policy violations per [`docs/agents/curation-authority-map.md`](../../../docs/agents/curation-authority-map.md) (#2473): do **not** reject `social-signal` rows merely for being first-party (`sameAsAuthor = 1.0`) or published within 30 days (`publishCooldownDays = 0` — no 30-day hold); instead flag `social-signal` rows that fail target-skill relevance (unrelated channel uploads or generic promo), lack provenance (`creatorRelation` / `relevanceNote`), fall below the 1,000-view scoring floor (`views < 1000` = 0 TM), or violate `(type, url)` dedup / 3-row same-creator plateau. Flag any `proxy-containment` row pointing to a first-party, canonical, or `suiteRef` repository or with `externalStars < 10000`. Flag any synthetic `self-attestation` row added solely to push a suite component past the 49.9 TM shared-repository baseline cap.
- Benchmark catalog misuse after Phase 2B: `benchmark-result` rows citing unknown, candidate, registered, rejected, or retired sources as if they score; vendor claims presented as reproducible benchmarks; scoring provenance on non-verified catalog entries; missing reproducibility fields or missing/dubious percentile values.
- Stale migration notes that still treat `tier_*.md` as the semantic working set.
- Multi-target peer-review packet misuse: wrong `evidenceType`, empty `targets`, invalid `skillId`, duplicate `(source.url, skillId, evidenceType)` rows, or forbidden strength/scoring fields (`trustNumber`, `grade`, `class`, `tier`, `level`, `stars`, `rank`).
- Imposter over-purging & audit taxonomy conflation (RFC #1809): ensure legitimate early-stage, self-hosted, or `installable: false` skills (Tier 2 Packaging Gap, Tier 3 Under-Evidenced Stub) are not mislabeled as Tier 1 Malicious Imposters or flagged for deletion.

Append concise findings to the source report. Do not mutate registry files.

## Optional Semantic Evidence Review Sidecar

An operator may optionally run the read-only Jev advisory runner to flag semantic anomalies in evidence rows ([docs/agents/jev.md](../../../docs/agents/jev.md)):

```bash
# Offline dry-run by default ($0 spend); add --live to query TypeSafe API
python scripts/jev_advisory.py --mode evidence --input <evidence-rows.json>
```

This output serves as review triage hints only. It never marks evidence rows verified, does not replace deterministic HTTP health or Phase 2B benchmark verification, does not touch Trust Magnitude scoring, and emits an actionable handoff to `worker-luna` on uncertainty.
rea-analysis-optimizationmorluto/reaOptimize REA providers and shared analysis workflows for agent context, latency, CPU, and memory while preserving correctness and Evidence. Use for analysis bottlenecks and cache or query design; CI/test-suite optimization belongs to rea-ci-tests-audit.Stars: 472kUpdated Claude CodeCodexVerifiedInstall
rea-ci-tests-auditmorluto/reaAudit REA CI and test suites for correctness, fixture drift, false passes, and measured performance; use for suite or workflow investigations and efficiency changes, rather than ordinary feature testing or a single test failure.Stars: 472kUpdated Claude CodeCodexVerifiedInstall
rea-tool-designmorluto/reaAudit, investigate, design, or change REA investigation tools, CLI/MCP contracts, provider capabilities, and Evidence semantics.Stars: 472kUpdated Claude CodeCodexVerifiedInstall
ponytail-auditDietrichGebert/ponytailQuality audit of the whole repo: bugs, security, real load, missing tests, speed, and what to delete. Most important first.Stars: 194kUpdated Claude CodeCodexVerifiedInstall