majiayu000/spellbook / skills/review-agent-harness
review-agent-harness
VerifiedReview whether a repository's coding-agent harness can reliably carry work from intent through controlled execution, verification, delivery, and learning. Use when asked to assess agent readiness, repeated agent failures, Rules/Skills/Hooks/Memory effectiveness, missing validation or recovery loops, or whether a harness repair improved later outcomes. Do not use for code-only audits, AGENTS-only audits, individual skill reliability reviews, or executing the task itself.
| File | Kind | Size |
|---|---|---|
| SKILL.md | skill md | 13.7 KB |
| agents/openai.yaml | config | 264 B |
| evals/evals.json | config | 7.8 KB |
| references/finding-contract.md | markdown | 6.8 KB |
| references/privacy-boundary.md | markdown | 2.2 KB |
| references/repair-loop.md | markdown | 2.9 KB |
| references/review-model.md | markdown | 4.9 KB |
| references/session-adapters.md | markdown | 2.5 KB |
| scripts/collect_evidence.py | script | 21.9 KB |
| scripts/harness_common.py | script | 14.1 KB |
| scripts/render_report.py | script | 19.9 KB |
| scripts/session_adapters.py | script | 16.6 KB |
| scripts/update_ledger.py | script | 15.1 KB |
| scripts/validate_findings.py | script | 29.6 KB |
references/repair-loop.md
View on GitHub# Repair Loop
Keep three claims separate:
1. **Finding exists**: reviewed evidence supports the causal problem.
2. **Repair verified**: a targeted check passes on the repaired final state.
3. **Later effect supported**: a comparable later Episode uses the route and
improves the outcome without a guardrail regression.
## Finding-Bound Repair
Lock the pre-fix finding id, consequence, owner, evidence, repair route, and
verifier. Route implementation to the smallest owner skill or project workflow.
Do not broaden the repair because neighboring improvements look convenient.
After the edit, run the exact verifier or a justified equivalent on the final
state. Set `repair_state` to:
- `verified` when the target-owned check passes;
- `partial` when only part of the causal chain is repaired;
- `blocked` when the named precondition remains unavailable;
- `not_started` before implementation.
Repair verification does not delete the finding or rewrite its original
severity.
## Longitudinal Effect
Update `learning-retention` to `outcome_supported` only when a later comparable
Episode shows all of these:
- the repaired mechanism was selected through its supported route;
- the task actually exercised it;
- the relevant result improved;
- safety, scope, and delivery guardrails did not regress.
Same-window reruns, configuration presence, Skill installation, and repair
completion are not later outcomes.
For a tool-backed repair, declare `--mechanism-category edit` or `validation`
while collecting both Episodes. Label them `--episode-role baseline` and
`--episode-role later` with the same explicit `--comparison-basis`. The matching
adapter count establishes only coarse exercise; bounded file or policy evidence
must map that category to the repaired route. Establish improvement and
non-regression with target-owned command or artifact references. Tool counts,
failure counts, request summaries, and collection timestamps are never
sufficient outcome evidence by themselves.
## Ledger Safety
Stable ids survive line movement, severity changes, and file renames when the
causal owner remains the same. The ledger marks an omitted open finding
`recheck_required`; only a targeted spot-check may confirm resolution. A
resolution-confirmations document must retain the verifier and bounded evidence
reference. Its verifier must exactly match the verifier locked into the first
ledger entry; changing the verifier requires a new finding identity or an
explicit review of the causal contract. A confirmed resolved id that reappears
is a regression.
```json
{
"schema_version": 1,
"kind": "agent-harness-resolution-confirmations",
"confirmations": [{
"id": "verification-closure--test-owner--final-state-not-rechecked",
"verifier": "The targeted final-state check passed.",
"evidence_ref": {
"kind": "command",
"locator": "targeted-test-final-state",
"claim": "The mapped check passed after the final edit."
}
}]
}
```
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| rea-analysis-optimizationmorluto/reaOptimize REA providers and shared analysis workflows for agent context, latency, CPU, and memory while preserving correctness and Evidence. Use for analysis bottlenecks and cache or query design; CI/test-suite optimization belongs to rea-ci-tests-audit. | Stars: 472k | Updated | Claude CodeCodex | Verified | Install |
| rea-ci-tests-auditmorluto/reaAudit REA CI and test suites for correctness, fixture drift, false passes, and measured performance; use for suite or workflow investigations and efficiency changes, rather than ordinary feature testing or a single test failure. | Stars: 472k | Updated | Claude CodeCodex | Verified | Install |
| rea-tool-designmorluto/reaAudit, investigate, design, or change REA investigation tools, CLI/MCP contracts, provider capabilities, and Evidence semantics. | Stars: 472k | Updated | Claude CodeCodex | Verified | Install |
| ponytail-auditDietrichGebert/ponytailQuality audit of the whole repo: bugs, security, real load, missing tests, speed, and what to delete. Most important first. | Stars: 194k | Updated | Claude CodeCodex | Verified | Install |