Skip to content

alirezarezvani/claude-skills / engineering/skills/agent-regression-evals

agent-regression-evals

Verified

Use before shipping any agent, prompt, model or tool change, or when a pass-rate drop needs a verdict. Runs repeated trials, gates candidate vs baseline with paired statistics in regression_gate.py, checks tool-call trajectories and finds flaky cases.

references/trajectory_grading.md

View on GitHub
# Grading the path, not only the answer

When `trajectory_checker.py` is worth running, which match mode to pick, and
how it fits beside outcome graders.

Confidence key: 🟒 verified on the primary source Β· 🟑 secondary Β· πŸ”΄ convention.

## 1. Grade the outcome first

Anthropic's guidance is that it is "often better to grade what the agent
produced, not the path it took", and that strict step-sequence checks are "too
rigid": a valid alternative path fails them. 🟒 [Anthropic]
tau-bench grades outcome as the final database state matching the ground-truth
state, plus required information appearing in the reply. 🟒 [tau-bench]

So the default eval is an outcome check. Add a trajectory check only where the
path itself is a requirement:

| Path requirement | Example | Mode |
|---|---|---|
| A safety boundary | never call `delete_order` while answering a FAQ | any mode + `forbidden` |
| A required lookup | must read the refund policy before refunding | `superset` |
| An order that matters | read before write; auth before data access | `strict` |
| A closed tool set | the agent may only use these three tools | `subset` |
| Runaway loops | more than 12 calls is a bug | `max_calls` |

## 2. The four match modes

The names follow LangChain's `agentevals`, the most widely used open
implementation. 🟒 [agentevals]

- `strict` β€” same calls, same order, nothing extra.
- `unordered` β€” same multiset of calls, any order.
- `subset` β€” every actual call appears in the reference (no extras; skipping is allowed).
- `superset` β€” every reference call appears in the actual run (extras allowed).

Argument matching: `exact`, `subset` (listed keys must match, extra keys fine;
the default here) or `ignore`. agentevals defaults to `exact`; this skill
defaults to `subset` because IDs and timestamps in extra arguments make exact
matching brittle. πŸ”΄ design choice.

## 3. Forbidden tools are not a match mode

A forbidden call blocks (exit 4) whatever the match mode and whatever the final
answer. A refund issued while answering a shipping question is a failure even
if the shipping answer was perfect. This is the cheapest high-value check in
the skill: one line per case.

## 4. Graders, in order of preference

1. **Code** β€” deterministic, cheap, reproducible; brittle to valid variation. 🟒 [Anthropic]
2. **Model** β€” flexible; non-deterministic, so calibrate against human labels before trusting it. 🟒 [Anthropic]
3. **Human** β€” the gold standard; slow, so spend it on calibrating 2 and on disputed cases. 🟒 [Anthropic]

Every script in this skill is a code grader. If your outcome grader is a model,
treat its disagreement with itself as a source of flakiness (see the flaky
cases reference) and run it on the same trial twice before blaming the agent.

## 5. Read the transcripts

"You won't know if your graders are working well unless you read the
transcripts." 🟒 [Anthropic] Before trusting a new trajectory spec, read five
passing and five failing trials by hand. A spec that fails valid paths teaches
the team to ignore the gate.

## 6. Where the cases come from

Start with 20-50 tasks drawn from real failures, balanced between what the
agent should do and what it should refuse to do. 🟒 [Anthropic] Regression
suites sit near 100%; capability suites start low and "graduate" into the
regression suite once saturated. 🟒 [Anthropic] SWE-bench's FAIL_TO_PASS /
PASS_TO_PASS split is the same idea applied to code: the fix must make the new
test pass and keep every old test passing. 🟑 [SWE-bench]

## Sources

1. Anthropic Engineering. *Demystifying evals for AI agents.* β€” https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents
2. Yao, S. et al. (2024). *Ο„-bench.* arXiv:2406.12045 β€” https://arxiv.org/abs/2406.12045
3. LangChain. *agentevals* (trajectory match evaluators). β€” https://github.com/langchain-ai/agentevals
4. Jimenez, C. E. et al. (2024). *SWE-bench: Can Language Models Resolve Real-World GitHub Issues?* ICLR 2024 β€” https://www.swebench.com/original.html
5. UK AI Security Institute. *Inspect* evaluation framework. β€” https://inspect.ai-safety-institute.org.uk/
6. OpenAI. *Evals* framework and registry. β€” https://github.com/openai/evals
reverse-engineer-anythingmorluto/reaReverse engineer native, managed, Electron/JavaScript, packaged, firmware, and browser targets with REA, and build agent environments (AgentEnv bundles) of web applications from a live instance or a capture, and of Linux desktop apps with a task from a request. Use shipped-artifact or requested runtime evidence to explain features, compare versions, decompile code, or guide a reconstruction. Skip REA for ordinary source-repository architecture analysis.Stars: 472kUpdated Claude CodeCodexVerifiedInstall
diagnosing-superpowersobra/superpowersUse when a superpowers session went wrong and your human partner wants to know why β€” repeated work, ignored plans, stumbles, poor results, a skill that didn't fire, "it took too long", "why is it so expensive", "what is it doing" β€” or wants to build a bug report for the superpowers maintainers, for the current session or a past one identified by id or path, on any harness.Stars: 322kUpdated Claude CodeCodexVerifiedInstall
dispatching-parallel-agentsobra/superpowersUse when facing 2+ independent tasks that can be worked on without shared state or sequential dependenciesStars: 322kUpdated Claude CodeCodexVerifiedInstall
executing-plansobra/superpowersUse when executing an implementation plan in the current session as the implementer yourself β€” your human partner chose inline execution, or no subagent tool is availableStars: 322kUpdated Claude CodeCodexVerifiedInstall