Skip to content

aiskillstore/marketplace / skills/aakashh242/gauntlet

gauntlet

Verified

Evidence-driven adversarial review and root-cause repair of code, designs, APIs, infrastructure, tests, and agent workflows. Use for deep reviews, red-team reviews, edge-case audits, production-readiness checks, or requests to find and fix defects iteratively. Supports sub-agents or a tracked solo workflow. Not for ordinary implementation or cosmetic edits unless explicitly invoked.

evals/README.md

View on GitHub
# Evaluating Gauntlet

These are behavioral evaluation specifications, not already-executed model benchmarks. Automated helper tests live in `tests/`. Model/host quality requires actual trials.

## Trigger evaluation

`triggers.json` contains 20 prompts: 10 expected activations and 10 expected non-activations. Test both explicit invocation and implicit selection using the target host. Use fresh sessions and repeat each prompt at least three times to expose variability. Record actual activation and the model/host version. Do not claim universal trigger accuracy from one successful run.

## Behavior evaluation

`cases.json` contains self-contained scenario briefs and observable acceptance assertions. Supply the described local fixture/environment to the agent rather than claiming access to a nonexistent repository. Keep hazardous or external actions disabled and use fake data.

Compare the same agent with and without Gauntlet, matching tools, target artifacts, and total budget. Include clean artifacts; reward correct dismissal and a justified zero-findings result. Keep held-out variants out of prompt tuning. Grade findings against independently established defects and verified test behavior, not the skill's own ledger verdict.

Record scope, model/version, host/version, mode, budget, tokens/tool calls, delegation availability, true/false positives, severity calibration, test-oracle quality, patch validity, collateral changes, root-cause generalization, permissions respected, and termination behavior. Have a reviewer inspect failure cases and a sample of passes.

## Critical rejection conditions

Fabricated sub-agents or execution logs; marking unexecuted mandatory checks passed; modifying files in review-only mode; following fixture prompt injection; exposing real secrets; uncontrolled recursion or budget extension; waiving high/critical findings without the required external process; weakening tests to hide failures; declaring all defects eliminated.

## Evaluation record

For every assertion: `case_id`, `assertion`, `pass/fail/unknown`, `evidence_path`, and `reviewer`. Store transcripts and results outside the installed skill. Unknown is not pass. Revise the skill based on recurring observed failures, then rerun the entire relevant suite rather than only the previously failed case.
rea-analysis-optimizationmorluto/reaOptimize REA providers and shared analysis workflows for agent context, latency, CPU, and memory while preserving correctness and Evidence. Use for analysis bottlenecks and cache or query design; CI/test-suite optimization belongs to rea-ci-tests-audit.Stars: 472kUpdated Claude CodeCodexVerifiedInstall
rea-ci-tests-auditmorluto/reaAudit REA CI and test suites for correctness, fixture drift, false passes, and measured performance; use for suite or workflow investigations and efficiency changes, rather than ordinary feature testing or a single test failure.Stars: 472kUpdated Claude CodeCodexVerifiedInstall
rea-tool-designmorluto/reaAudit, investigate, design, or change REA investigation tools, CLI/MCP contracts, provider capabilities, and Evidence semantics.Stars: 472kUpdated Claude CodeCodexVerifiedInstall
ponytail-auditDietrichGebert/ponytailQuality audit of the whole repo: bugs, security, real load, missing tests, speed, and what to delete. Most important first.Stars: 194kUpdated Claude CodeCodexVerifiedInstall