borghei/Claude-Skills / engineering/agent-harness
agent-harness
UnverifiedTest and evaluation harness for AI agents — scenario suites, deterministic replay, regression diffing, cost and latency budgets. Use when agent quality is vibe-checked, before shipping a prompt or model change, or when evals drift.
| File | Kind | Size |
|---|---|---|
| SKILL.md | skill md | 14.4 KB |
| assets/eval_report_template.md | markdown | 2.0 KB |
| assets/sample_baseline_report.json | config | 7.9 KB |
| assets/sample_candidate_report.json | config | 7.9 KB |
| assets/sample_suite.json | config | 3.2 KB |
| assets/sample_transcripts_baseline.json | config | 3.0 KB |
| assets/sample_transcripts_candidate.json | config | 2.8 KB |
| assets/scenario_authoring_checklist.md | markdown | 2.4 KB |
| references/eval-methodology-and-budgets.md | markdown | 9.5 KB |
| references/scenario-and-fixture-design.md | markdown | 9.8 KB |
| scripts/eval_diff.py | script | 10.0 KB |
| scripts/scenario_runner.py | script | 11.3 KB |
assets/scenario_authoring_checklist.md
View on GitHub# Scenario Authoring Checklist
Run through this before adding a scenario to a gating suite. A scenario that
fails any of the first five items will cost more in maintenance than it returns
in signal.
## The scenario is complete
- [ ] **Input** — the user turn(s), verbatim
- [ ] **World state** — what the tools will find, pinned (including an `as_of`
date if correctness depends on time)
- [ ] **Tool behaviour** — recorded results for every tool the agent may call,
including the error cases
- [ ] **Assertions** — at least one structural assertion, not only text matching
- [ ] **Budget** — latency, cost, and turn ceiling (or explicit reliance on the
suite defaults)
## The assertions are durable
- [ ] No assertion quotes a full sentence of model output
- [ ] Text assertions target domain tokens (IDs, policy names, numbers), not phrasing
- [ ] At least one `tool_not_called` assertion if the agent can take an
irreversible action in this scenario
- [ ] Every assertion carries a severity: `critical` / `major` / `minor`
- [ ] `critical` is reserved for safety, money, data loss, and refusals that
must hold — not for "the answer was a bit worse"
## The scenario earns its place
- [ ] It covers a bucket the suite is thin on (happy / boundary / refusal /
adversarial / failure-recovery / ambiguity)
- [ ] It came from a real incident, a real customer report, or a real
disagreement about intended behaviour — not from imagination
- [ ] It fails against the *previous* known-bad behaviour (verify by running it
against the pre-fix transcript; a scenario that passes both before and
after tests nothing)
- [ ] Its runtime is acceptable for the suite tier it is joining (core suites
should stay under 5 minutes end to end)
## Fixture hygiene
- [ ] All PII replaced with stable synthetic values at record time
- [ ] `prompt_sha` and `model` recorded in the run file
- [ ] No secrets, tokens, or internal URLs in the recorded tool results
- [ ] Fixture committed to version control and reviewed like code
## Before it becomes a gate
- [ ] The scenario has passed on two consecutive identical-config runs (a
scenario that flips on a no-op is flaky, not a gate)
- [ ] The owning team is named and knows they will be paged when it fails
- [ ] There is a documented answer to "what do we do when this fails" beyond
"investigate"
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| finishing-a-development-branchobra/superpowersUse when implementation is complete, all tests pass, and you need to decide how to integrate the work | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |
| systematic-debuggingobra/superpowersUse when encountering any bug, test failure, or unexpected behavior, before proposing fixes | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |
| test-driven-developmentobra/superpowersUse when implementing any feature or bugfix, before writing implementation code | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |
| skill-creatoranthropics/skillsCreate new skills, modify and improve existing skills, and measure skill performance. Use when users want to create a skill from scratch, edit, or optimize an existing skill, run evals to test a skill, benchmark skill performance with variance analysis, or optimize a skill's description for better triggering accuracy. | Stars: 198k | Updated | Claude CodeCodex | Verified | Install |