alirezarezvani/claude-skills / engineering/skills/agent-regression-evals
agent-regression-evals
VerifiedUse before shipping any agent, prompt, model or tool change, or when a pass-rate drop needs a verdict. Runs repeated trials, gates candidate vs baseline with paired statistics in regression_gate.py, checks tool-call trajectories and finds flaky cases.
| File | Kind | Size |
|---|---|---|
| SKILL.md | skill md | 5.3 KB |
| assets/ci_eval_gate.example.yml | config | 1.7 KB |
| assets/eval_case_spec.example.json | config | 669 B |
| assets/sample_baseline.jsonl | other | 3.7 KB |
| assets/sample_candidate.jsonl | other | 3.7 KB |
| references/eval_statistics.md | markdown | 4.8 KB |
| references/flaky_cases_and_quarantine.md | markdown | 4.8 KB |
| references/trajectory_grading.md | markdown | 4.1 KB |
| scripts/flake_detector.py | script | 9.8 KB |
| scripts/regression_gate.py | script | 12.2 KB |
| scripts/trajectory_checker.py | script | 10.6 KB |
references/trajectory_grading.md
View on GitHub# Grading the path, not only the answer
When `trajectory_checker.py` is worth running, which match mode to pick, and
how it fits beside outcome graders.
Confidence key: π’ verified on the primary source Β· π‘ secondary Β· π΄ convention.
## 1. Grade the outcome first
Anthropic's guidance is that it is "often better to grade what the agent
produced, not the path it took", and that strict step-sequence checks are "too
rigid": a valid alternative path fails them. π’ [Anthropic]
tau-bench grades outcome as the final database state matching the ground-truth
state, plus required information appearing in the reply. π’ [tau-bench]
So the default eval is an outcome check. Add a trajectory check only where the
path itself is a requirement:
| Path requirement | Example | Mode |
|---|---|---|
| A safety boundary | never call `delete_order` while answering a FAQ | any mode + `forbidden` |
| A required lookup | must read the refund policy before refunding | `superset` |
| An order that matters | read before write; auth before data access | `strict` |
| A closed tool set | the agent may only use these three tools | `subset` |
| Runaway loops | more than 12 calls is a bug | `max_calls` |
## 2. The four match modes
The names follow LangChain's `agentevals`, the most widely used open
implementation. π’ [agentevals]
- `strict` β same calls, same order, nothing extra.
- `unordered` β same multiset of calls, any order.
- `subset` β every actual call appears in the reference (no extras; skipping is allowed).
- `superset` β every reference call appears in the actual run (extras allowed).
Argument matching: `exact`, `subset` (listed keys must match, extra keys fine;
the default here) or `ignore`. agentevals defaults to `exact`; this skill
defaults to `subset` because IDs and timestamps in extra arguments make exact
matching brittle. π΄ design choice.
## 3. Forbidden tools are not a match mode
A forbidden call blocks (exit 4) whatever the match mode and whatever the final
answer. A refund issued while answering a shipping question is a failure even
if the shipping answer was perfect. This is the cheapest high-value check in
the skill: one line per case.
## 4. Graders, in order of preference
1. **Code** β deterministic, cheap, reproducible; brittle to valid variation. π’ [Anthropic]
2. **Model** β flexible; non-deterministic, so calibrate against human labels before trusting it. π’ [Anthropic]
3. **Human** β the gold standard; slow, so spend it on calibrating 2 and on disputed cases. π’ [Anthropic]
Every script in this skill is a code grader. If your outcome grader is a model,
treat its disagreement with itself as a source of flakiness (see the flaky
cases reference) and run it on the same trial twice before blaming the agent.
## 5. Read the transcripts
"You won't know if your graders are working well unless you read the
transcripts." π’ [Anthropic] Before trusting a new trajectory spec, read five
passing and five failing trials by hand. A spec that fails valid paths teaches
the team to ignore the gate.
## 6. Where the cases come from
Start with 20-50 tasks drawn from real failures, balanced between what the
agent should do and what it should refuse to do. π’ [Anthropic] Regression
suites sit near 100%; capability suites start low and "graduate" into the
regression suite once saturated. π’ [Anthropic] SWE-bench's FAIL_TO_PASS /
PASS_TO_PASS split is the same idea applied to code: the fix must make the new
test pass and keep every old test passing. π‘ [SWE-bench]
## Sources
1. Anthropic Engineering. *Demystifying evals for AI agents.* β https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents
2. Yao, S. et al. (2024). *Ο-bench.* arXiv:2406.12045 β https://arxiv.org/abs/2406.12045
3. LangChain. *agentevals* (trajectory match evaluators). β https://github.com/langchain-ai/agentevals
4. Jimenez, C. E. et al. (2024). *SWE-bench: Can Language Models Resolve Real-World GitHub Issues?* ICLR 2024 β https://www.swebench.com/original.html
5. UK AI Security Institute. *Inspect* evaluation framework. β https://inspect.ai-safety-institute.org.uk/
6. OpenAI. *Evals* framework and registry. β https://github.com/openai/evals
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| reverse-engineer-anythingmorluto/reaReverse engineer native, managed, Electron/JavaScript, packaged, firmware, and browser targets with REA, and build agent environments (AgentEnv bundles) of web applications from a live instance or a capture, and of Linux desktop apps with a task from a request. Use shipped-artifact or requested runtime evidence to explain features, compare versions, decompile code, or guide a reconstruction. Skip REA for ordinary source-repository architecture analysis. | Stars: 472k | Updated | Claude CodeCodex | Verified | Install |
| diagnosing-superpowersobra/superpowersUse when a superpowers session went wrong and your human partner wants to know why β repeated work, ignored plans, stumbles, poor results, a skill that didn't fire, "it took too long", "why is it so expensive", "what is it doing" β or wants to build a bug report for the superpowers maintainers, for the current session or a past one identified by id or path, on any harness. | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |
| dispatching-parallel-agentsobra/superpowersUse when facing 2+ independent tasks that can be worked on without shared state or sequential dependencies | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |
| executing-plansobra/superpowersUse when executing an implementation plan in the current session as the implementer yourself β your human partner chose inline execution, or no subagent tool is available | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |