Skip to content

gaia-research/gaia-skill-tree / .agents/skills/ev-benchmark-verification

ev-benchmark-verification

Verified

Phase 2B of the Gaia evidence verification pipeline. Verifies benchmark-result rows and benchmark-source candidates against registry/benchmark-sources.json before adversarial audit, link validation, catalog promotion, or ingestion. Use when running /ev-pipeline, checking benchmark-result evidence, reviewing benchmark-source candidates, or preparing benchmark rows for /gaia-ingest-batch.

FileKindSize
SKILL.mdskill md3.8 KB
---
name: ev-benchmark-verification
description: >
  Phase 2B of the Gaia evidence verification pipeline. Verifies benchmark-result rows and benchmark-source candidates against registry/benchmark-sources.json before adversarial audit, link validation, catalog promotion, or ingestion. Use when running /ev-pipeline, checking benchmark-result evidence, reviewing benchmark-source candidates, or preparing benchmark rows for /gaia-ingest-batch.
---

# Benchmark-Source Verification (ev-benchmark-verification)

Phase 2B runs after `/ev-star-verification` and before `/ev-adversarial-audit`. It is benchmark-only and read-only: it classifies existing `benchmark-result` registry rows and optional benchmark-source candidate manifests without mutating the registry, the catalog, or the evidence lake.

## Policy

`registry/benchmark-sources.json` is the benchmark allow/blacklist surface. Current lanes collapse to three values:

- `verified` — CI-reproduced or verifier-attested benchmark evidence; Trust Magnitude lane multiplier `2.0x`.
- `reported` — public claim or mirrored leaderboard evidence accepted by the human gate; Trust Magnitude lane multiplier `1.0x`.
- `rejected` — blacklisted, disputed, pending, retired, candidate, unknown, or otherwise not approved; Trust Magnitude multiplier `0x`.

Legacy aliases are accepted during migration: `ci-reproduced` and `verifier-attested` normalize to `verified`; `mirrored` normalizes to `reported`; `pending`, `unknown`, `retired`, and `candidate` normalize to `rejected`/no scoring. A catalog entry with `status: rejected` is the blacklist and always scores zero.

Reported rows do not need `runAt`, `datasetHash`, or `benchmarkInputHash`. Verified rows still need reproducibility fields. `appliesToGenericSkillRefs` records which generic skill IDs a benchmark applies to; it does not create a named-skill score by itself.

## Command

```bash
PYTHONPATH=src python evidence/scripts/verify_benchmark_sources.py \
  --catalog registry/benchmark-sources.json \
  --registry-root . \
  --check \
  --report /tmp/ev-phase2b-benchmark-report.md
```

With candidate sources discovered during Phase 0 or issue review:

```bash
PYTHONPATH=src python evidence/scripts/verify_benchmark_sources.py \
  --catalog registry/benchmark-sources.json \
  --registry-root . \
  --candidate-manifest /tmp/benchmark-candidates.jsonl \
  --check \
  --report /tmp/ev-phase2b-benchmark-report.md
```

Candidate manifest entries may be JSON or JSONL objects with fields such as:

```json
{
  "target": "firecrawl/firecrawl-research-index",
  "source": "https://www.firecrawl.dev/blog/research-index-launch",
  "benchmarkId": "alphaxiv-arxivqa@v1.0",
  "status": "candidate",
  "missing": [],
  "notes": "Candidate becomes reported only after the human gate approves it."
}
```

Firecrawl Research Index / alphaXiv ArXivQA is the model case: a public reported benchmark can count at 1.0x after the human gate, but a candidate report alone is not permission to ingest.

## Human gate

Machines classify. Humans approve or reject.

After Phase 2B benchmark-source verification, Phase 3 adversarial audit, and Phase 4 link validation, stop before benchmark catalog promotion or `/gaia-ingest-batch`. A human reviewer must approve reported benchmark evidence or mark it rejected/blacklisted. Do not treat a green Phase 2B report as permission to ingest.

## Output

The report summarizes:

- `scoring-eligible` rows that may count toward Trust Magnitude.
- `reported`, `rejected-row`, `candidate-only`, `blocked`, and `unknown-benchmark` rows that need human review or score zero.
- rejected/retired usage and scoring-lane misuse as hard blockers.
- missing verified-lane reproducibility fields and malformed percentiles as findings.

Candidate-only manifest entries are never hard blockers by themselves; they are backlog items for human review.
rea-analysis-optimizationmorluto/reaOptimize REA providers and shared analysis workflows for agent context, latency, CPU, and memory while preserving correctness and Evidence. Use for analysis bottlenecks and cache or query design; CI/test-suite optimization belongs to rea-ci-tests-audit.Stars: 472kUpdated Claude CodeCodexVerifiedInstall
rea-ci-tests-auditmorluto/reaAudit REA CI and test suites for correctness, fixture drift, false passes, and measured performance; use for suite or workflow investigations and efficiency changes, rather than ordinary feature testing or a single test failure.Stars: 472kUpdated Claude CodeCodexVerifiedInstall
rea-tool-designmorluto/reaAudit, investigate, design, or change REA investigation tools, CLI/MCP contracts, provider capabilities, and Evidence semantics.Stars: 472kUpdated Claude CodeCodexVerifiedInstall
ponytail-auditDietrichGebert/ponytailQuality audit of the whole repo: bugs, security, real load, missing tests, speed, and what to delete. Most important first.Stars: 194kUpdated Claude CodeCodexVerifiedInstall