Skip to content

charlieviettq/awesome-agent-skill / .claude/skills/agent-evaluation

agent-evaluation

Verified

Evaluate LLM agents and tool-using workflows—task success, tool accuracy, latency/cost, safety, and regression suites. Use when shipping agent features, comparing prompts/models, or debugging agent failures.

Agent evaluation

What to measure

DimensionExamples
Task successEnd state matches spec (binary or rubric)
Tool useCorrect tool, valid args, no spurious calls
SafetyNo policy violations, no secret leakage
EfficiencyTokens, latency, tool call count
StabilitySame input -> consistent outcome across runs

Workflow

  1. Define tasks — realistic user intents with clear pass/fail or scored rubric.
  2. Build dataset — golden set + edge cases (errors, ambiguous input, empty context).
  3. Run baseline — fixed model/settings; log traces (inputs, tools, outputs).
  4. Score — automated checks first; human review for ambiguous cases.
  5. Compare — A/B prompts, models, or tool schemas; report deltas with confidence notes.
  6. Gate — block release on regression in must-pass tasks.

Automated checks

  • Schema validation on tool arguments.
  • Assert final answer contains required fields or avoids forbidden content.
  • Snapshot tests for deterministic sub-steps where possible.

Human rubric (when needed)

Score 1-5 on: correctness, completeness, tone, safety. Document disagreements.

Anti-patterns

  • Eval only on cherry-picked happy paths.
  • Changing task and model simultaneously without isolation.
  • No trace logs when debugging tool failures.

Output

Summary table: variant | success rate | avg tools | avg latency | notes.

reverse-engineer-anythingmorluto/reaReverse engineer native, managed, Electron/JavaScript, packaged, firmware, and browser targets with REA, and build agent environments (AgentEnv bundles) of web applications from a live instance or a capture, and of Linux desktop apps with a task from a request. Use shipped-artifact or requested runtime evidence to explain features, compare versions, decompile code, or guide a reconstruction. Skip REA for ordinary source-repository architecture analysis.Stars: 471kUpdated Claude CodeCodexVerifiedInstall
diagnosing-superpowersobra/superpowersUse when a superpowers session went wrong and your human partner wants to know why — repeated work, ignored plans, stumbles, poor results, a skill that didn't fire, "it took too long", "why is it so expensive", "what is it doing" — or wants to build a bug report for the superpowers maintainers, for the current session or a past one identified by id or path, on any harness.Stars: 322kUpdated Claude CodeCodexVerifiedInstall
dispatching-parallel-agentsobra/superpowersUse when facing 2+ independent tasks that can be worked on without shared state or sequential dependenciesStars: 322kUpdated Claude CodeCodexVerifiedInstall
executing-plansobra/superpowersUse when executing an implementation plan in the current session as the implementer yourself — your human partner chose inline execution, or no subagent tool is availableStars: 322kUpdated Claude CodeCodexVerifiedInstall