JPeetz/agent-skills / skills/qa-testing/agent-evaluation
agent-evaluation
UnverifiedUse this skill when building systematic evaluation for LLM agents and agent outputs — defining quality dimensions, curating golden datasets, running LLM-as-judge, pairwise and rubric-based evaluations, tracking regressions, and gating CI on quality. Covers promptfoo, DeepEval, LangSmith, Inspect, and lm-evaluation-harness, with judge calibration, cost/latency budgets, and eval-driven development for prompt, model, and tool changes. Not for deterministic unit tests of application code (see test-driven-development), tracing/dashboards only (see llm-observability), or evaluator-optimizer architecture (see agentic-eval).
| File | Kind | Size |
|---|---|---|
| CHANGELOG.md | markdown | 3.1 KB |
| LICENSE | other | 1.0 KB |
| SKILL.md | skill md | 25.7 KB |
| evals/evals.json | config | 4.3 KB |
| references/evaluation-metrics.md | markdown | 4.2 KB |
| references/judge-design.md | markdown | 4.6 KB |
| references/tooling.md | markdown | 7.1 KB |
| scripts/eval_gate.py | script | 7.3 KB |
references/evaluation-metrics.md
View on GitHub# Evaluation Metrics Reference
## Dimension → metric → evaluator
| Dimension | Metric | Evaluator | Score range | Typical gate |
|---|---|---|---|---|
| Correctness (closed) | exact match / normalized match | code | 0/1 | ≥ 0.95 |
| Correctness (open) | answer accuracy | judge vs reference | 0–1 | floor 0.85, non-inferior |
| Groundedness | faithfulness | RAG judge over context | 0–1 | floor 0.90 |
| Retrieval quality | contextual recall / precision | RAG judge with reference | 0–1 | diagnostic, not gate |
| Task success (agent) | completion rate | environment check or trajectory judge | 0–1 | floor 0.85 |
| Tool selection | tool correctness | trace compare vs expected tools | 0–1 | ≥ 0.95 |
| Tool arguments | argument correctness | judge over args + input | 0–1 | ≥ 0.90 |
| Efficiency | step efficiency / tool calls per task | count from trace | int | ≤ baseline × 1.25 |
| Safety | violation rate | classifier + judge | 0–1 | exactly 0 |
| Instruction adherence | schema/format pass rate | code | 0/1 | exactly 1.0 |
| Helpfulness | pairwise win rate vs baseline | pairwise judge | 0–1 | ≥ 0.50 |
| Tone/policy | rubric pass rate | rubric judge | 0–1 | floor 0.90 |
| Cost | USD per request/task | usage × price | float | ≤ budget |
| Latency | p50/p95 ms | timer on request | ms | ≤ budget |
## RAG metric definitions (DeepEval / Ragas vocabulary)
| Metric | Question it answers | Needs |
|---|---|---|
| **Answer relevancy** | Does the answer address the question asked? | input, actual_output |
| **Faithfulness** | Is every claim supported by retrieved context? | actual_output, retrieval_context |
| **Contextual precision** | Are the most relevant chunks ranked first? | input, retrieval_context, expected_output |
| **Contextual recall** | Did retrieval fetch everything needed? | input, retrieval_context, expected_output |
| **Hallucination** | Does output contradict supplied context? | actual_output, context |
| **Answer correctness** | Does output match the reference answer? | actual_output, expected_output |
Faithfulness and answer relevancy are gate material. Contextual precision/recall
are diagnostic: when faithfulness drops, they tell you whether the retriever or
the generator is at fault.
## Agent trajectory metrics
| Metric | Definition | Failure it catches |
|---|---|---|
| Task completion | End state satisfies the user's goal | Right text, no action taken |
| Tool correctness | Called tools ⊆ expected tools | Wrong tool, hallucinated tool |
| Argument correctness | Arguments match the values in the input | Refund for wrong order/amount |
| Step efficiency | Steps used vs minimum needed | Loops, redundant retries |
| Plan adherence | Actions consistent with stated plan | Silent scope drift |
Agents need at least task completion and one tool metric. Output-only scoring
marks an agent that issued three duplicate refunds as correct.
## Threshold guidance
1. **Derive floors from the current accepted system**, not from aspiration. Run
the suite on the deployed version, take the mean, set the floor 2–3 points
below it, then raise it as the system improves.
2. **Zero-tolerance only for binary policy facts** (PII leak, forbidden action,
schema violation). Applying zero tolerance to a judge metric guarantees a
flaky gate.
3. **Non-inferiority beats absolute improvement** for iteration: `new >= old -
tolerance`. Absolute-only gates either block every change or get lowered
until they mean nothing.
4. **Judge metrics need wider tolerances than code metrics** — 3–5 points
versus 0–1 — because judge variance is real.
5. **Cost/latency budgets are ratios to baseline** (+10% cost, +20% p95) unless
the product has a hard SLA, in which case use the SLA.
## Reporting template
```text
suite=v3 n=214 pass=0.882 CI95=(0.833,0.919) baseline=0.897
slice billing/refund n=48 pass=0.833 CI95=(0.704,0.913) Δ -0.062 <-- regressed
slice account/security n=31 pass=0.935 CI95=(0.793,0.982) Δ +0.010
slice general n=135 pass=0.889 CI95=(0.825,0.933) Δ -0.004
cost usd/req=0.0024 (+9.1%) p95=3120ms (+4.0%)
GATE: FAIL (G3 billing/refund)
```
The delta and the interval, side by side, are what make the report honest. A
slice with `n=12` should be printed but excluded from gating.
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| finishing-a-development-branchobra/superpowersUse when implementation is complete, all tests pass, and you need to decide how to integrate the work | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |
| systematic-debuggingobra/superpowersUse when encountering any bug, test failure, or unexpected behavior, before proposing fixes | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |
| test-driven-developmentobra/superpowersUse when implementing any feature or bugfix, before writing implementation code | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |
| skill-creatoranthropics/skillsCreate new skills, modify and improve existing skills, and measure skill performance. Use when users want to create a skill from scratch, edit, or optimize an existing skill, run evals to test a skill, benchmark skill performance with variance analysis, or optimize a skill's description for better triggering accuracy. | Stars: 198k | Updated | Claude CodeCodex | Verified | Install |