Skip to content

borghei/Claude-Skills / engineering/agent-harness

agent-harness

Unverified

Test and evaluation harness for AI agents — scenario suites, deterministic replay, regression diffing, cost and latency budgets. Use when agent quality is vibe-checked, before shipping a prompt or model change, or when evals drift.

assets/eval_report_template.md

View on GitHub
# Agent Eval Report — <suite name>

**Candidate:** <run id> (<model / prompt sha>)
**Baseline:** <run id> (<model / prompt sha>)
**Date:** YYYY-MM-DD
**Decision requested:** ship / hold / ship with waiver

---

## Verdict

<One sentence. Lead with the decision, not the numbers.>

| Signal | Baseline | Candidate | Movement |
|--------|----------|-----------|----------|
| Critical failures | | | |
| Scenarios regressed | — | | |
| Scenarios fixed | — | | |
| Pass rate (95% CI) | | | |
| Total cost | | | |
| Mean latency | | | |

---

## Critical failures

> Any entry here blocks release. If empty, write "None."

| Scenario | Assertion | What the agent did | Blast radius |
|----------|-----------|--------------------|--------------|
| | | | |

---

## Regressions

| Scenario | Failing assertions | Suspected cause | Owner |
|----------|--------------------|-----------------|-------|
| | | | |

## Fixes

| Scenario | Previously failing | Confirmed by |
|----------|--------------------|--------------|
| | | |

---

## Budget drift

> Scenarios whose cost or latency moved more than the drift threshold, even
> where both runs stayed inside budget.

| Scenario | Cost Δ | Latency Δ | Cause |
|----------|--------|-----------|-------|
| | | | |

---

## Statistical read

- Discordant scenarios (flipped either direction): **<b + c>**
- Exact McNemar p: **<p>**
- Interpretation: <If fewer than ~6 flipped, state that the p-value is not
  informative at this size and that the decision rests on the individual
  regressions above.>
- Noise floor: <flips observed when the identical config was diffed against
  itself, or "not measured" — and if not measured, say so.>

---

## Waivers requested

| Scenario | Why the regression is acceptable | Approver | Expires |
|----------|----------------------------------|----------|---------|
| | | | |

---

## Follow-ups

- [ ] New scenarios added from this run's findings
- [ ] Budgets re-derived from the accepted run
- [ ] Baseline report promoted to `release-current.report.json`
finishing-a-development-branchobra/superpowersUse when implementation is complete, all tests pass, and you need to decide how to integrate the workStars: 322kUpdated Claude CodeCodexVerifiedInstall
systematic-debuggingobra/superpowersUse when encountering any bug, test failure, or unexpected behavior, before proposing fixesStars: 322kUpdated Claude CodeCodexVerifiedInstall
test-driven-developmentobra/superpowersUse when implementing any feature or bugfix, before writing implementation codeStars: 322kUpdated Claude CodeCodexVerifiedInstall
skill-creatoranthropics/skillsCreate new skills, modify and improve existing skills, and measure skill performance. Use when users want to create a skill from scratch, edit, or optimize an existing skill, run evals to test a skill, benchmark skill performance with variance analysis, or optimize a skill's description for better triggering accuracy.Stars: 198kUpdated Claude CodeCodexVerifiedInstall