majiayu000/spellbook / skills/review-agent-harness
review-agent-harness
VerifiedReview whether a repository's coding-agent harness can reliably carry work from intent through controlled execution, verification, delivery, and learning. Use when asked to assess agent readiness, repeated agent failures, Rules/Skills/Hooks/Memory effectiveness, missing validation or recovery loops, or whether a harness repair improved later outcomes. Do not use for code-only audits, AGENTS-only audits, individual skill reliability reviews, or executing the task itself.
| File | Kind | Size |
|---|---|---|
| SKILL.md | skill md | 13.7 KB |
| agents/openai.yaml | config | 264 B |
| evals/evals.json | config | 7.8 KB |
| references/finding-contract.md | markdown | 6.8 KB |
| references/privacy-boundary.md | markdown | 2.2 KB |
| references/repair-loop.md | markdown | 2.9 KB |
| references/review-model.md | markdown | 4.9 KB |
| references/session-adapters.md | markdown | 2.5 KB |
| scripts/collect_evidence.py | script | 21.9 KB |
| scripts/harness_common.py | script | 14.1 KB |
| scripts/render_report.py | script | 19.9 KB |
| scripts/session_adapters.py | script | 16.6 KB |
| scripts/update_ledger.py | script | 15.1 KB |
| scripts/validate_findings.py | script | 29.6 KB |
references/session-adapters.md
View on GitHub# Session Adapters
The MVP supports Codex and Claude Code JSONL through
`scripts/collect_evidence.py`. Both adapters consume only explicit files or
explicit roots and emit the same privacy-safe facts envelope.
## Codex
The adapter recognizes `response_item` message, function-call, custom-tool-call,
and tool-output records. It counts user turns, tool calls, edit calls,
validation calls, failures, malformed lines, and unsupported lines. Structured
exit codes and explicit status fields take precedence over prose; benign text
such as `failed tests: 0` is not a failure. It never emits call ids, arguments,
raw outputs, or source paths.
## Claude Code
The adapter recognizes `user` and `assistant` message records, `tool_use`
blocks, and error-marked `tool_result` blocks. It emits the same normalized
counts as the Codex adapter and never emits tool-use ids or raw inputs.
## Selection
- Use one provider per evidence envelope.
- Prefer exact `--session-file` values.
- Use `--session-root` only for an explicitly authorized recursive boundary.
- The root adapter caps discovery with `--max-session-files`; omitted files
remain counted.
- Each file is also bounded by bytes, lines, line bytes, and retained warnings.
Defaults are 16 MiB, 100,000 lines, 1 MiB per line, and 200 warnings. A hit
emits `status: constrained`, `input_truncated`, and explicit truncation
reasons; test fixtures may lower the bounds with matching
`--max-session-*` flags.
- Use `--include-request-summaries` only when the decision requires task intent.
These adapters are evidence collectors, not outcome classifiers. Counts and
sanitized summaries route review; the lead still decides whether an observed
Episode supports a finding or dimension state.
For tool-backed later-effect review, `--mechanism-category edit` or
`validation` declares the coarse count being examined. Pair `--episode-role`
with an explicit `--comparison-basis` so baseline and later envelopes state the
comparison instead of inferring it from collection time. The envelope does not
claim that the mechanism caused a result; route mapping and target-owned result
evidence remain required.
## Unsupported Or Drifted Input
Malformed JSONL increments `malformed_lines`; valid but unknown records increment
`unsupported_lines`. Both produce anonymous warnings. A source with no
recognized records is `unobserved`, not present. If a provider changes its
record shape, update synthetic fixtures and patch the adapter before trusting
the new source.
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| rea-analysis-optimizationmorluto/reaOptimize REA providers and shared analysis workflows for agent context, latency, CPU, and memory while preserving correctness and Evidence. Use for analysis bottlenecks and cache or query design; CI/test-suite optimization belongs to rea-ci-tests-audit. | Stars: 472k | Updated | Claude CodeCodex | Verified | Install |
| rea-ci-tests-auditmorluto/reaAudit REA CI and test suites for correctness, fixture drift, false passes, and measured performance; use for suite or workflow investigations and efficiency changes, rather than ordinary feature testing or a single test failure. | Stars: 472k | Updated | Claude CodeCodex | Verified | Install |
| rea-tool-designmorluto/reaAudit, investigate, design, or change REA investigation tools, CLI/MCP contracts, provider capabilities, and Evidence semantics. | Stars: 472k | Updated | Claude CodeCodex | Verified | Install |
| ponytail-auditDietrichGebert/ponytailQuality audit of the whole repo: bugs, security, real load, missing tests, speed, and what to delete. Most important first. | Stars: 194k | Updated | Claude CodeCodex | Verified | Install |