dvy1987/agent-loom / .agents/skills/eval-pipeline
eval-pipeline
VerifiedDesign automated evaluation pipelines for LLM and agent systems — combining deterministic checks, statistical metrics, and LLM-as-judge scoring into repeatable, CI-integrated eval suites. Load when the user asks to set up automated evals, design an eval pipeline, integrate evals into CI/CD, create an eval suite, do eval-driven development, or says "automate my evals", "CI eval integration", "evaluation pipeline", "continuous evaluation", "monitoring eval quality", "set up regression testing for my agent". Sub-skill of eval-output orchestrator.
| File | Kind | Size |
|---|---|---|
| README.md | markdown | 625 B |
| SKILL.md | skill md | 11.0 KB |
| references/examples.md | markdown | 2.2 KB |
| references/harness-regression.md | markdown | 2.2 KB |
references/harness-regression.md
View on GitHub# Harness Regression Mode
Invoked when `harness-evolution` Step 4 runs. Distinct from general LLM eval pipelines.
## Metric: pass@1 with k-rollouts (AHE)
- Run **k ≥ 2** independent rollouts per task (default k=3 under noise).
- **pass@1** = mean binary success over all k × |D| attempts — not pass@k best-of.
- Report per-split: held-in, held-out separately.
## Dual-split acceptance (Self-Harness)
Promote candidate only if **all** hold:
| Rule | Condition |
|------|-----------|
| Held-in | Δ ≥ 0 vs parent harness |
| Held-out | Δ ≥ 0 vs parent harness |
| Strict gain | max(Δ) > 0 — not tie on both splits |
**Trade-off trap:** Reject edits where held-in gains but held-out regresses (or vice versa) even if total pass count rises.
## Stochastic aggregation
When tasks are noisy:
1. Repeat full candidate evaluation (same k per task).
2. Aggregate pass counts across repeats before applying dual-split rule.
3. Single lucky rollout must not promote.
## Manifest prediction telemetry (AHE)
Track per round in `change_manifest.json`:
| Field | Use |
|-------|-----|
| `predicted_impact.fixes` | Tasks expected to pass after edit |
| `predicted_impact.risk_tasks` | Tasks expected to regress — **low trust** (~2× random) |
After eval, log precision/recall separately for fix vs regression predictions. Do **not** skip held-out gate because regressions were "predicted."
## Regression suite gate (auto-harness adapted)
| Step | Check |
|------|-------|
| 0 | File guard — no edits outside `allowed_write_paths` |
| 1 | Suite `docs/harness/suite.json` pass rate ≥ threshold |
| 2 | Full held-out pass@1 ≥ best prior in `docs/harness/results.tsv` |
| 3 | Promote newly passing held-in tasks into suite (expand coverage) |
## Pareto selection (Meta-Harness)
When optimizing multiple objectives (accuracy × context-cost × latency):
- Return **frontier** of non-dominated candidates — not single scalar winner.
- Final held-out eval on frontier points only; proposer never sees held-out results.
## Label-free fallback (RHO)
Weaker evidence path — see `harness-evolution/references/evolution-loop.md`. Prefer this harness regression mode when labeled eval exists.
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| finishing-a-development-branchobra/superpowersUse when implementation is complete, all tests pass, and you need to decide how to integrate the work | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |
| systematic-debuggingobra/superpowersUse when encountering any bug, test failure, or unexpected behavior, before proposing fixes | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |
| test-driven-developmentobra/superpowersUse when implementing any feature or bugfix, before writing implementation code | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |
| skill-creatoranthropics/skillsCreate new skills, modify and improve existing skills, and measure skill performance. Use when users want to create a skill from scratch, edit, or optimize an existing skill, run evals to test a skill, benchmark skill performance with variance analysis, or optimize a skill's description for better triggering accuracy. | Stars: 198k | Updated | Claude CodeCodex | Verified | Install |