Skip to content

alirezarezvani/claude-skills / engineering/skills/agent-regression-evals

agent-regression-evals

Verified

Use before shipping any agent, prompt, model or tool change, or when a pass-rate drop needs a verdict. Runs repeated trials, gates candidate vs baseline with paired statistics in regression_gate.py, checks tool-call trajectories and finds flaky cases.

Agent Regression Evals — prove the change did not make it worse

Portability: 3 stdlib scripts, no APIs, no LLM calls. You run the agent; these grade, count and gate.

"Pass rate went from 82% to 79%" is not a finding until you know whether three points is outside the noise. Agents are non-deterministic, so one trial per case cannot tell a regression from a coin flip, and a final answer can be right while the path to it called a tool it never should have.

Workflow

# 0 - Run every case k times (k >= 3) on baseline and candidate, from a clean
#     environment per trial. Write one JSONL row per trial:
#     {"case_id": "refund-lookup", "trial": 1, "passed": true, "critical": true,
#      "cost_usd": 0.012, "trajectory": [{"tool": "get_order", "args": {...}}]}
# 1 - Grade the path where the path matters (safety, required lookups).
python scripts/trajectory_checker.py --spec cases.json --results candidate.jsonl
# 2 - Separate flaky cases from stable ones before trusting any comparison.
python scripts/flake_detector.py --results candidate.jsonl --k 3 --quarantine-out q.json
# 3 - Gate: paired per-case comparison against baseline. Exit 4 blocks the merge.
python scripts/regression_gate.py --baseline baseline.jsonl --candidate candidate.jsonl --quarantine q.json

Step 1 uses match modes strict | unordered | subset | superset. Default to superset (required steps present, harmless exploration allowed); grade the outcome, not the path, unless order is the requirement. forbidden tools block in every mode.

Step 2 reports pass@k (succeeds at least once in k) and pass^k (succeeds all k times). A user meets every trial, so pass^k is the reliability number. Cases marked critical are never quarantine candidates.

Step 3 averages trials within each case first (trials of one case are correlated), then tests paired per-case differences with a t-based 95% CI and a one-sided exact McNemar test. When the drop is real but the sample is too small to show it, the gate says how many cases it would take.

The gate blocks on

  1. A statistically significant drop (CI upper bound below zero, or McNemar p < 0.05).
  2. A critical case that passed every baseline trial and failed any candidate trial.
  3. A baseline case missing from the candidate run. Dropping a case is the cheapest fake fix.
  4. A quarantine entry with no owner or expiry, or covering a critical case.
  5. A forbidden tool call (trajectory checker, exit 4).

It warns, and a human decides, on an underpowered drop above --tolerance or a cost rise above --max-cost-increase.

Hard rules

  • Never compare single trials. Under 3 trials per case the flake detector refuses (exit 4).
  • Never pool trials as independent samples. Average per case, then compare cases.
  • Never quarantine without owner and expiry. An expired entry re-enters the gate automatically.
  • Never delete a failing case to go green. The gate counts what is missing.
  • Never call "no significant drop" a pass on 8 cases. Report the power note with the verdict.
  • Start the suite from real failures (20-50 cases), balanced between "should do" and "should not do".

Scripts

ScriptRoleExit codes
scripts/trajectory_checker.pyGrades tool-call paths: match modes, args modes, forbidden tools, call ceilings0 · 2 deviations · 3 bad input · 4 forbidden tool
scripts/flake_detector.pySTABLE_PASS / STABLE_FAIL / FLAKY per case, Wilson CI, pass@k, pass^k, quarantine stubs0 · 2 flaky · 3 · 4 too few trials
scripts/regression_gate.pyPaired CI, exact McNemar, critical cases, dropped cases, quarantine validity, cost0 PASS · 2 WARN · 3 · 4 BLOCK

All support --sample and --output json.

References and assets

reverse-engineer-anythingmorluto/reaReverse engineer native, managed, Electron/JavaScript, packaged, firmware, and browser targets with REA, and build agent environments (AgentEnv bundles) of web applications from a live instance or a capture, and of Linux desktop apps with a task from a request. Use shipped-artifact or requested runtime evidence to explain features, compare versions, decompile code, or guide a reconstruction. Skip REA for ordinary source-repository architecture analysis.Stars: 471kUpdated Claude CodeCodexVerifiedInstall
diagnosing-superpowersobra/superpowersUse when a superpowers session went wrong and your human partner wants to know why — repeated work, ignored plans, stumbles, poor results, a skill that didn't fire, "it took too long", "why is it so expensive", "what is it doing" — or wants to build a bug report for the superpowers maintainers, for the current session or a past one identified by id or path, on any harness.Stars: 322kUpdated Claude CodeCodexVerifiedInstall
dispatching-parallel-agentsobra/superpowersUse when facing 2+ independent tasks that can be worked on without shared state or sequential dependenciesStars: 322kUpdated Claude CodeCodexVerifiedInstall
executing-plansobra/superpowersUse when executing an implementation plan in the current session as the implementer yourself — your human partner chose inline execution, or no subagent tool is availableStars: 322kUpdated Claude CodeCodexVerifiedInstall