majiayu000/spellbook / skills/review-agent-harness
review-agent-harness
VerifiedReview whether a repository's coding-agent harness can reliably carry work from intent through controlled execution, verification, delivery, and learning. Use when asked to assess agent readiness, repeated agent failures, Rules/Skills/Hooks/Memory effectiveness, missing validation or recovery loops, or whether a harness repair improved later outcomes. Do not use for code-only audits, AGENTS-only audits, individual skill reliability reviews, or executing the task itself.
| File | Kind | Size |
|---|---|---|
| SKILL.md | skill md | 13.7 KB |
| agents/openai.yaml | config | 264 B |
| evals/evals.json | config | 7.8 KB |
| references/finding-contract.md | markdown | 6.8 KB |
| references/privacy-boundary.md | markdown | 2.2 KB |
| references/repair-loop.md | markdown | 2.9 KB |
| references/review-model.md | markdown | 4.9 KB |
| references/session-adapters.md | markdown | 2.5 KB |
| scripts/collect_evidence.py | script | 21.9 KB |
| scripts/harness_common.py | script | 14.1 KB |
| scripts/render_report.py | script | 19.9 KB |
| scripts/session_adapters.py | script | 16.6 KB |
| scripts/update_ledger.py | script | 15.1 KB |
| scripts/validate_findings.py | script | 29.6 KB |
references/finding-contract.md
View on GitHub# Finding Contract
## Contents
- [Eligibility](#eligibility)
- [Stable Identity](#stable-identity)
- [Required JSON Shape](#required-json-shape)
- [Reconciliation](#reconciliation)
- [Verification](#verification)
## Eligibility
Promote a candidate only when it has all of these:
1. An observed consequence or exact violated governing requirement.
2. A bounded file, command, artifact, policy, runtime, or Session-fact
reference.
3. A causal explanation and smallest owner.
4. A repair route using discovered project or provider capabilities.
5. A verifier that can distinguish repaired from unrepaired state.
Do not promote counts, search absence without a requirement, similarity,
unavailable evidence, theoretical risk, stale reports, or inventory presence.
## Stable Identity
Build `id` from primary dimension, smallest owner, and root-cause phrase:
```text
verification-closure--test-owner--final-state-not-rechecked
```
Use lowercase slugs with at least three `--`-separated components. Never put a
line number, date, severity, Session id, or transient branch name in the id.
## Required JSON Shape
```json
{
"schema_version": 1,
"kind": "agent-harness-findings",
"overview": "The harness exposes task routes but does not close final-state verification.",
"scope": {
"target": "example-project",
"target_id": "local-sha256:<64 hex characters from the collector>",
"snapshot": {
"baseline": "current_checkout",
"target_relation": "exact_git_root",
"id": "git-sha256:<64 hex characters from the collector>"
},
"mode": "static",
"locale": "en",
"providers": ["none"],
"decision": "Assess whether the harness closes the task loop.",
"acceptance_boundary": "Resolve all five dimensions with bounded evidence states.",
"output_mode": "durable"
},
"evidence_boundary": {
"included": ["repository-static-evidence"],
"excluded": ["user-home-discovery", "memory-bodies", "raw-transcripts", "secret-values", "stable-session-identifiers"],
"unavailable": ["session-evidence:not_authorized"]
},
"dimensions": [
{
"id": "task-contract",
"status": "healthy",
"evidence_state": "reachable",
"confidence": "medium",
"score": 80,
"score_rationale": "All three checks have reachable project-owned routes.",
"summary": "The repository routes tasks to scoped instructions.",
"evidence_refs": [
{
"kind": "file",
"locator": "AGENTS.md:1",
"claim": "The root file declares the task routing boundary."
}
]
}
],
"checks": [
{
"id": "goal-understanding",
"dimension": "task-contract",
"status": "healthy",
"evidence_state": "reachable",
"confidence": "medium",
"summary": "The task contract retains a recoverable goal.",
"evidence_refs": [
{
"kind": "file",
"locator": "AGENTS.md:1",
"claim": "The root contract defines the task boundary."
}
],
"finding_refs": []
}
],
"verification_runs": [
{
"id": "targeted-test",
"purpose": "targeted_reproduction",
"result": "supports",
"exit_code": 1,
"final_state": true,
"summary": "The final-state reproduction confirms the validation gap."
}
],
"findings": [
{
"id": "verification-closure--test-owner--final-state-not-rechecked",
"title": "The final edit was not rechecked",
"severity": "high",
"confidence": "high",
"primary_dimension": "verification-closure",
"primary_check": "validate-again",
"evidence_state": "exercised",
"consequence": "Completion was reported without validation on the final state.",
"root_cause": "The mapped check ran before the last material edit.",
"owner": "project test workflow",
"evidence_refs": [
{
"kind": "command",
"locator": "targeted-test",
"claim": "The retained pass predates the final edit."
}
],
"repair_route": "Run the mapped check on the final state.",
"verifier": "The mapped check passes after the last material edit.",
"verification_state": "confirmed",
"repair_state": "not_started"
}
],
"priority_moves": [
"verification-closure--test-owner--final-state-not-rechecked"
]
}
```
The real document must contain all five dimensions exactly once. Valid values
are enforced by `scripts/validate_findings.py`. It must also contain all 15
stable checks exactly once. Each finding names one `primary_check`, and that
check reverse-links the finding id through `finding_refs`.
Copy `target_id` and the full `snapshot` object from the same collector
envelope used for the review. They bind durable output to one concrete local
directory and its frozen state; do not invent, reuse, or edit these digests.
Each dimension requires `score` and `score_rationale`. The score cannot exceed
the weakest applicable check's evidence ceiling from
[Review Model](review-model.md), and no overall score is permitted. A numeric
score never establishes finding eligibility.
Every executed verifier belongs in `verification_runs`. Critical and High
findings may use `confirmed` only when a command evidence locator matches a
final-state `candidate_refutation` or `targeted_reproduction` run whose result
is `supports`. Preserve both a failing child/subcheck and the aggregate exit
code when diagnosing false-green test wrappers.
For `learning-retention: outcome_supported`, evidence references must cover five
unique purposes: `baseline_episode`, `later_episode`, `route_mapping`,
`outcome_check`, and `guardrail_check`. Baseline and later references use
`kind: session_fact` plus the same `comparison_basis` and
`mechanism_category`. Route mapping uses `file` or `policy`; result and guardrail
checks use `command`, `artifact`, or `runtime`.
The validator also requires exactly two collector-produced evidence envelopes
for this state. They must bind to the same longitudinal findings scope, contain
unique baseline/later roles, share comparison basis and mechanism category, use
a named supported provider, and show the later mechanism was exercised.
## Reconciliation
Keep findings separate when consequence, root cause, owner, or verifier differ.
Merge only duplicate evidence for the same causal problem. Do not delete an
eligible finding to reach a presentation target. Rank at most three ids in
`priority_moves`.
## Verification
Actively try to refute Critical and High findings by opening the named owner and
its caller, guard, config, or final output. Use `confirmed` only when the claim
survives. Use `unverified` or `unavailable` otherwise and keep that limitation
visible. A verifier result is evidence, not permission to repair. Prior reports
are leads only; repeat the mapped check against `scope.snapshot` before using
them as current evidence.
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| rea-analysis-optimizationmorluto/reaOptimize REA providers and shared analysis workflows for agent context, latency, CPU, and memory while preserving correctness and Evidence. Use for analysis bottlenecks and cache or query design; CI/test-suite optimization belongs to rea-ci-tests-audit. | Stars: 472k | Updated | Claude CodeCodex | Verified | Install |
| rea-ci-tests-auditmorluto/reaAudit REA CI and test suites for correctness, fixture drift, false passes, and measured performance; use for suite or workflow investigations and efficiency changes, rather than ordinary feature testing or a single test failure. | Stars: 472k | Updated | Claude CodeCodex | Verified | Install |
| rea-tool-designmorluto/reaAudit, investigate, design, or change REA investigation tools, CLI/MCP contracts, provider capabilities, and Evidence semantics. | Stars: 472k | Updated | Claude CodeCodex | Verified | Install |
| ponytail-auditDietrichGebert/ponytailQuality audit of the whole repo: bugs, security, real load, missing tests, speed, and what to delete. Most important first. | Stars: 194k | Updated | Claude CodeCodex | Verified | Install |