Skip to content

ranbot-ai/awesome-skills / skills/ai-agent-evaluation-benchmarking

ai-agent-evaluation-benchmarking

Verified

AI Agent Capability Evaluation & Benchmarking

What it is: Standardizes multi-metric capability benchmarking, token cost efficiency, and regression monitoring across autonomous coding agents.

Overview

Provides a standardized, auditable framework and data model for AI Agent Capability Evaluation & Benchmarking operations across distributed engineering and decentralized application systems.

When to Use This Skill

  • When recording agent evaluation runs with a fixed dataset, rubric, runtime/model revision and measured outcomes.
  • When comparing a candidate with a baseline under matched tasks, environment, tool permissions and cost accounting.
  • When generating reproducible CSV, SQL DDL, JSON Schema, and Notion property registers for tracking compliance.

How It Works

  1. Define the parameters, thresholds, and identity bindings required for the target operational register.
  2. Select appropriate boundary enforcement values from validated enum select sets.
  3. Export standardized artifacts (CSV table, SQL DDL, JSON Schema) to integrate into validation CI pipelines.

Field Reference

#Field NameTypeSQL TypeJSON Schema TypeNotion Property TypeExample Value
1Benchmark Run IDidSERIAL PRIMARY KEYintegerText1
2Evaluated Agent ModelselectVARCHAR(64)stringSelectClaude 3.7 Sonnet
3Benchmark Suite DomainselectVARCHAR(64)stringSelectSWE-bench Verified
4Tasks Evaluated CountnumberINTEGERintegerNumber100
5Pass Rate PercentagenumberNUMERIC(5,2)numberNumber78.40
6Tool Hallucination Rate %numberNUMERIC(5,2)numberNumber0.60
7Average Tokens Per TasknumberINTEGERintegerNumber42500
8Cost Per Solved Task USDcurrencyNUMERIC(8,4)numberNumber0.3420
9Regression VerdictselectVARCHAR(32)stringSelectImproved with evidence
10Evaluation LeadtextVARCHAR(64)stringTextRanjeet2063
11Benchmark Execution DatedateDATEstring, format: dateDate2026-10-01

Select Options

Evaluated Agent Model

Use the exact evaluated model/provider/deployment revision; examples are not an availability allowlist.

Benchmark Suite Domain

SWE-bench Verified | WebArena | AgentBench | HumanEval-Rust | Web3AuditBench

Regression Verdict

Improved with evidence | No demonstrated change | Regressed with evidence | Inconclusive

Measurement procedure

Record an immutable run identifier, suite revision, task IDs, model/runtime/tool settings, completed/failed/skipped counts and actual charges. Derive pass rate from eligible completed outcomes under a declared denominator; tool hallucination needs a defined invalid-call denominator. For a regression verdict, use paired outcomes or a justified test and interval, and retain an inconclusive state. Labels and example values below are synthetic, not measured results. Define any relation field in every emitted format only when the target exists and is confirmed.

Relations

  • Audit Reference -> links to the formal review documentation or test repository.
  • Target Architecture -> links to the deployed contract or autonomous agent runtime component.

Examples

Prompt

How do I configure and track AI Agent Capability Evaluation & Benchmarking for our production environment?

Recommended Next Step

Generate the unified field schema, SQL DDL migration, and JSON validation schema to register into your system catalog.

Workflow: Define criteria -> Run automated verification -> Record baseline -> Monitor invariants.

Best Practices

  • Enforce strict typing on numerical bounds and currency amounts; avoid unstructured free-text fields for critical states.
  • Re-run validation test suites on every state-altering commit or parameter change.
  • Keep example data synthetic and isolated from production cryptographic keys or private endpoints.

Limitations

  • This register does not execute benchmarks, prove compliance or need on-chain RPC credentials. An actual authorized harness and evaluation evidence are external prerequisites.
  • Scores are comparable only with matched suite/revision, eligibility, failure/timeout handling and sampling. Record counts and uncertainty, not merely a percentage.
  • Define cost per solved task as total attributable run cost divided by solved tasks; zero solves means unavailable/infinite cost under an explicit policy, not zero. Token estimates alone omit provider/tool charges.

Security & Safety Notes

  • Generating a register is distinct from running paid providers, transmitting private tasks, changing CI or deploying; those effects need actual authorization.
  • Use synthetic dummy
reverse-engineer-anythingmorluto/reaReverse engineer native, managed, Electron/JavaScript, packaged, firmware, and browser targets with REA, and build agent environments (AgentEnv bundles) of web applications from a live instance or a capture, and of Linux desktop apps with a task from a request. Use shipped-artifact or requested runtime evidence to explain features, compare versions, decompile code, or guide a reconstruction. Skip REA for ordinary source-repository architecture analysis.Stars: 471kUpdated Claude CodeCodexVerifiedInstall
diagnosing-superpowersobra/superpowersUse when a superpowers session went wrong and your human partner wants to know why — repeated work, ignored plans, stumbles, poor results, a skill that didn't fire, "it took too long", "why is it so expensive", "what is it doing" — or wants to build a bug report for the superpowers maintainers, for the current session or a past one identified by id or path, on any harness.Stars: 322kUpdated Claude CodeCodexVerifiedInstall
dispatching-parallel-agentsobra/superpowersUse when facing 2+ independent tasks that can be worked on without shared state or sequential dependenciesStars: 322kUpdated Claude CodeCodexVerifiedInstall
executing-plansobra/superpowersUse when executing an implementation plan in the current session as the implementer yourself — your human partner chose inline execution, or no subagent tool is availableStars: 322kUpdated Claude CodeCodexVerifiedInstall