Skip to content

gaia-research/gaia-skill-tree / .agents/skills/gaia-audit

gaia-audit

Verified

Focused single-target audit of one Gaia skill, named skill, or real-skill catalog item. Use this skill when: auditing a specific skill by ID or name, a skill looks overpromoted, evidence feels thin or dead, a contributor's skill seems outdated or superseded, the rank doesn't match the source quality, evidence is mapped to the wrong Evidence Type, a skill at 3★+ is missing a working GitHub link, someone asks "does this skill deserve its stars?", or you want the smallest possible correction with validation — not a bulk sweep. This is NOT for bulk curation passes (use /gaia-curate for that) or meta-level registry sweeps (use /gaia-meta-audit). It is specifically for "fix this one skill, get it right."

gaia-audit

Yggdrasil IV: Trust degradation and reversible technical-challenge contract. This focused audit verifies evidence, distinguishes temporary liveness from independently verified material counterevidence, and prepares a traceable adjudication packet. An automated HTTP check, model claim, inactivity, or competitor marketing cannot change TM/stars or declare a skill obsolete. Class B Steward can close proven technical challenges only, while canonical changes require Class C approval.

Perform a focused, single-target audit of one Gaia skill or catalog item and apply the smallest correct source-level fix. The goal is accuracy over coverage: one skill, done properly.

Why single-target?

Bulk passes introduce unintended side effects and make diffs hard to review. A single-target audit lets you go deep — verifying the actual source material, not just the metadata — and produces a minimal, reviewable change. If you find adjacent issues while auditing, note them but do not fix them in this pass.

Workflow

1. Locate the target

Determine which registry layer owns the skill:

  • Generic (starless) canonical skill: registry/nodes/<id>.json (or via registry/gaia.json)
  • Named skill implementation: registry/named/<contributor>/<skill>.md
  • Real-skill catalog item: registry/real-skills.json

Run rg <skill-id> across the repo to find every occurrence — frontmatter references, named files, catalog entries, and timeline events — before touching anything.

2. Ask the diagnostic questions

Work through these in order. Stop when you find a violation — that's the correction to make.

  • Does the skill exist at all in the registry?
  • Is it generic (no stars) or named (2★–6★)? Generic skills have no level; only named implementations carry stars.
  • Does it map to the claimed Gaia capability? (Check CONTEXT.md for canonical vocabulary.)
  • Does the evidence actually support the claimed rank (stars)? Re-read each source URL — homepage and directory listings are weak; SKILL.md blobs, papers, benchmarks, and release notes are strong.
  • Is it outdated, superseded, or a duplicate of something already in the registry?
  • For named skills at 3★+: is there an independently confirmed absence of the required blob/branch/subpath implementation link? A temporary dead URL is only an observation; a verified Star Bar violation may warrant an approved hard-reset review.
  • For fusion/upgrade opportunities: note if related basic skills could cleanly consolidate, but do not act on it in this pass unless the target is explicitly the consolidation candidate.
  • Does the skill have structural, documentation, or packaging issues? Classify against the Four-Tier Audit Taxonomy (GOVERNANCE.md §4.2 / RFC #1809):
    • Tier 1 (Malicious / Imposter Squatter): Bad-faith aggregation, hijacking other maintainers' repos/stars, or falsified attribution → Expungement from registry.
    • Tier 2 (Packaging / Install Shape Gap): Authentic concept and author, but missing standard packaging or broken links → For ≤2★, tag installable: false per CONTRIBUTING.md §12. For 3★+, independently establish the Star Bar facts, request repackaging, and queue the existing 1★ consequence for approved adjudication; do not demote from a single URL probe. Never purge or label as an imposter.
    • Tier 3 (Early-Stage / Under-Evidenced Stub): Authentic author, thin docs or preliminary evidence → Keep at baseline 1★ (Awakened), request doc enrichment, or tag needs-info. Never purge.
    • Tier 4 (Product-Coupled / Non-Generalized): Real tool documentation needing generalization (#1766, #1801) → Enforce maker attribution via check_product_attribution.py and route to generalization.
  • Mandatory Contributor Check: If the author is a known contributor, co-founder, or team member, deletion is strictly prohibited without direct maintainer outreach and documented review.

Skip the rarity field entirely — it is deprecated and carries no review signal (see CONTEXT.md § Rarity).

3. Present findings before acting

Write out what you found and the proposed correction before making any edit. This gives the user a chance to redirect. If no correction is warranted, say so explicitly with the reason — "no action" is a valid and useful audit outcome.

4. Apply the correction via CLI

All mutations go through the CLI, not direct file edits — this preserves timeline logging and schema integrity.

# Adjust rank (named skills only — use contributor/skill format, not bare generic IDs)
gaia dev calibrate <contributor/skill_id> <N>★

# Change generic type (basic / fusion)
gaia dev reclassify <generic_id> <type>

# Update named status (awakened / named / etc.)
gaia dev update-named <contributor/skill_id> --status <status>

# Add or replace evidence
gaia dev evidence <skill_id> "<url>" --type <arxiv|repo|github-stars|...> --trust <value>

# Log a demotion event in the timeline
gaia dev timeline <skill_id> --action demote --notes "<reason>"

For dead evidence links: create/refresh Steward verification debt and collect repeated source/provenance evidence. Do not remove evidence, mint demerits or write demotion events merely because a URL fails. After separately approved adjudication, use governed CLI mutations and complete both canonical and owner-tree timeline receipts.

5. Regenerate and verify

gaia dev build       # sync all derivative files after registry edits
gaia dev validate
gaia dev validate --intake
gaia dev test all
git diff --check

Demotion enforcement rules

These are hard rules — they apply regardless of how prominent or well-known the skill is:

  • 3★+ with independently confirmed missing implementation blob → propose the META.md §2.4 1★ hard-reset consequence through Class C adjudication; perform a canonical/owner-tree timeline-safe CLI change only after approval. A transient 404 is not sufficient.
  • Unique-branch skill demoted below 4★ → it loses its Unique-branch rank status; if its structure no longer warrants fusion, reclassify the generic to basic (gaia dev reclassify accepts only basic / fusion)
  • Dead evidence links → mark as liveness observation; investigate and surface verified challenge, never auto-remove evidence or auto-demote.
  • Placeholder or non-verifiable evidence only → present the evidence deficit and seek independent verification / approved canonical ruling; no automatic Awakened reset.
  • Suite components: the suite itself does not need links.github, but each component must have a specific blob/branch/subpath URL

Frontier challenger-lead input

A challenger-lead-v1.1, frontier-verification-request-v1, frontier-experiment-receipt-v1 or frontier-meta-handoff-v1 is an unverified request, never proof of independent verification. Read docs/architecture/ecosystem-intelligence.md; validate source bytes, exact actual target identity/revision/content digest, claim/campaign/protocol/fixture/history/request cross-bindings and dedupe. Execute only the fixed reviewed repository protocol; never execute packet strings, infer superiority from similarity/vendor marketing, or accept a self-declared actor/reviewer/receipt/status. Treat timeout/unavailable as execution_failed, not a negative result; treat comparative mismatch as inconclusive unless the fixed protocol's interpretation criteria support a narrow conclusion. Record both positive and counterevidence. No result here authenticates the verifier or admits a public challenge. Do not alter canonical evidence, TM, stars, Origin, or Steward state; guarded admission uses the existing challenge-domain trusted resolver and fails closed when unavailable.

Meta verification record

When the input includes an experiment receipt, finish by writing one frontier-meta-verification-v1 record (src/gaia_cli/data/frontier/meta-verification-v1.schema.json). It is a claim for human Meta review, never an approval or admission. Positive outcomes, rerun traces and triage-stage records use schemaVersion: frontier-meta-verification-v1.1; see gaia-frontier for the trace requirements.

  • Separate session. Verify in a different session from the one that produced the receipt. Set verifierRunId to a fresh id that differs from the receipt's experimentRunId, and bind requestId, receiptRunId and receiptDigest (frontier.receipt_digest) to the receipt you assessed.
  • Re-execute, don't re-read. Run the fixed reviewed protocol yourself and store the outputs under the case's evidence/frontier-challenges/<id>/artifacts/ (or generated-output/frontier/ while drafting). List them in reexecution.artifacts with digests, using a different path from every receipt output. substantiated/narrowed (and any request-admission-review recommendation) are refused unless the receipt result is supported with matching documented/executed versions, the artifacts cover every receipt output label, the native artifact parses under the request-pinned protocol (same experiment, request, fixture, protocol and executed version) and its output equals the receipt's native output after the protocol's declared outputNormalization (exact match by default; git-log-L masks commit ids), and none is a byte-identical copy of a receipt output. Because narrowed shares the supported-receipt requirement, against a contradicted, inconclusive or execution_failed receipt only contradicted, inconclusive or execution_failed outcomes are available; record any narrower finding in limitations and recommend narrow-claim or hold.
  • Counterevidence. State what you searched for in counterevidence.searched even when items is empty; list contrary findings with a source.
  • Outcome is one of substantiated, narrowed, contradicted, inconclusive, execution_failed. Outages and timeouts are execution_failed; a documented-versus-executed version mismatch can be at most inconclusive. recommend is a suggestion (request-admission-review only for substantiated or narrowed).
  • Independence is declared, not proven. Always "independence": "declared-unauthenticated"; reexecution.performed, verifierRunId and "separate session" are claims by the record's author that nothing here authenticates. Never add approval, admitted, verified, actor, signature, confidence, TM, stars, level or Origin fields; the validator refuses them at any depth.
  • Check the record with gaia dev frontier meta-validate --record <record> --receipt <receipt> --request <request> --artifact-root <artifact root> (--request is required; add --repo-root outside the checkout). A pass is a structural check only.

Output

Conclude with a concise report covering:

  • Which skill was audited and what layer it lives in
  • Evidence reviewed (URLs checked and their quality)
  • Correction applied, or the explicit reason no change was warranted
  • CLI commands run
  • Validation results
rea-analysis-optimizationmorluto/reaOptimize REA providers and shared analysis workflows for agent context, latency, CPU, and memory while preserving correctness and Evidence. Use for analysis bottlenecks and cache or query design; CI/test-suite optimization belongs to rea-ci-tests-audit.Stars: 471kUpdated Claude CodeCodexVerifiedInstall
rea-ci-tests-auditmorluto/reaAudit REA CI and test suites for correctness, fixture drift, false passes, and measured performance; use for suite or workflow investigations and efficiency changes, rather than ordinary feature testing or a single test failure.Stars: 471kUpdated Claude CodeCodexVerifiedInstall
rea-tool-designmorluto/reaAudit, investigate, design, or change REA investigation tools, CLI/MCP contracts, provider capabilities, and Evidence semantics.Stars: 471kUpdated Claude CodeCodexVerifiedInstall
ponytail-auditDietrichGebert/ponytailQuality audit of the whole repo: bugs, security, real load, missing tests, speed, and what to delete. Most important first.Stars: 194kUpdated Claude CodeCodexVerifiedInstall