borghei/Claude-Skills / engineering/agent-harness
agent-harness
UnverifiedTest and evaluation harness for AI agents — scenario suites, deterministic replay, regression diffing, cost and latency budgets. Use when agent quality is vibe-checked, before shipping a prompt or model change, or when evals drift.
| File | Kind | Size |
|---|---|---|
| SKILL.md | skill md | 14.4 KB |
| assets/eval_report_template.md | markdown | 2.0 KB |
| assets/sample_baseline_report.json | config | 7.9 KB |
| assets/sample_candidate_report.json | config | 7.9 KB |
| assets/sample_suite.json | config | 3.2 KB |
| assets/sample_transcripts_baseline.json | config | 3.0 KB |
| assets/sample_transcripts_candidate.json | config | 2.8 KB |
| assets/scenario_authoring_checklist.md | markdown | 2.4 KB |
| references/eval-methodology-and-budgets.md | markdown | 9.5 KB |
| references/scenario-and-fixture-design.md | markdown | 9.8 KB |
| scripts/eval_diff.py | script | 10.0 KB |
| scripts/scenario_runner.py | script | 11.3 KB |
assets/eval_report_template.md
View on GitHub# Agent Eval Report — <suite name>
**Candidate:** <run id> (<model / prompt sha>)
**Baseline:** <run id> (<model / prompt sha>)
**Date:** YYYY-MM-DD
**Decision requested:** ship / hold / ship with waiver
---
## Verdict
<One sentence. Lead with the decision, not the numbers.>
| Signal | Baseline | Candidate | Movement |
|--------|----------|-----------|----------|
| Critical failures | | | |
| Scenarios regressed | — | | |
| Scenarios fixed | — | | |
| Pass rate (95% CI) | | | |
| Total cost | | | |
| Mean latency | | | |
---
## Critical failures
> Any entry here blocks release. If empty, write "None."
| Scenario | Assertion | What the agent did | Blast radius |
|----------|-----------|--------------------|--------------|
| | | | |
---
## Regressions
| Scenario | Failing assertions | Suspected cause | Owner |
|----------|--------------------|-----------------|-------|
| | | | |
## Fixes
| Scenario | Previously failing | Confirmed by |
|----------|--------------------|--------------|
| | | |
---
## Budget drift
> Scenarios whose cost or latency moved more than the drift threshold, even
> where both runs stayed inside budget.
| Scenario | Cost Δ | Latency Δ | Cause |
|----------|--------|-----------|-------|
| | | | |
---
## Statistical read
- Discordant scenarios (flipped either direction): **<b + c>**
- Exact McNemar p: **<p>**
- Interpretation: <If fewer than ~6 flipped, state that the p-value is not
informative at this size and that the decision rests on the individual
regressions above.>
- Noise floor: <flips observed when the identical config was diffed against
itself, or "not measured" — and if not measured, say so.>
---
## Waivers requested
| Scenario | Why the regression is acceptable | Approver | Expires |
|----------|----------------------------------|----------|---------|
| | | | |
---
## Follow-ups
- [ ] New scenarios added from this run's findings
- [ ] Budgets re-derived from the accepted run
- [ ] Baseline report promoted to `release-current.report.json`
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| finishing-a-development-branchobra/superpowersUse when implementation is complete, all tests pass, and you need to decide how to integrate the work | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |
| systematic-debuggingobra/superpowersUse when encountering any bug, test failure, or unexpected behavior, before proposing fixes | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |
| test-driven-developmentobra/superpowersUse when implementing any feature or bugfix, before writing implementation code | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |
| skill-creatoranthropics/skillsCreate new skills, modify and improve existing skills, and measure skill performance. Use when users want to create a skill from scratch, edit, or optimize an existing skill, run evals to test a skill, benchmark skill performance with variance analysis, or optimize a skill's description for better triggering accuracy. | Stars: 198k | Updated | Claude CodeCodex | Verified | Install |