Skip to content

moonlight-lupin/agent-skills / plugins/dev-pipeline/skills/review-lessons

review-lessons

Verified

Library of lessons from past build-review loops — not a stage. Use when a review or fix round hits a specific case: a reviewer BLOCKING that contradicts the brief, a test that passes but checks nothing, a feature dead in production, SQLite/WAL migrations, MCP tool surfaces, REST API authz, app ↔ external-source sync, Android/APK verification, fill-template extraction, CLI/git mechanics, reviewer outages. Load the one file the case needs; never load the whole library.

FileKindSize
SKILL.mdexecutableskill md5.5 KB
references/architecture-refactor-test-updates.mdexecutablescript2.9 KB
references/beta-gate-release-hardening.mdexecutablescript3.3 KB
references/ci-and-repo-hygiene.mdexecutablescript3.7 KB
references/confirm-loop-lessons.mdexecutablescript24.2 KB
references/cross-model-parallel-review.mdexecutablescript4.8 KB
references/data-integrity-principles.mdexecutablescript7.3 KB
references/delegate-skills-adaptation.mdexecutablescript2.1 KB
references/evidence-pairs.mdexecutablescript2.5 KB
references/extraction-from-filled-docs.mdexecutablescript6.1 KB
references/field-briefing-integration.mdexecutablescript5.2 KB
references/full-module-replacement-scoping.mdexecutablescript6.3 KB
references/generated-frontmatter-and-reviewer-outages.mdexecutablescript3.0 KB
references/loop-pitfalls.mdexecutablescript27.9 KB
references/mcp-category-tool-implementation.mdexecutablescript5.3 KB
references/mcp-design-review-patterns.mdexecutablescript5.7 KB
references/minified-bundle-selector-research.mdexecutablescript4.4 KB
references/mobile-app-verification.mdexecutablescript4.8 KB
references/multi-batch-sequencing.mdexecutablescript5.7 KB
references/parallel-dual-review.mdexecutablescript10.8 KB
references/pitfalls-dispatch-semantics.mdexecutablescript6.3 KB
references/pitfalls-fix-round-verification.mdexecutablescript6.6 KB
references/planning-with-files-adaptation.mdexecutablescript2.6 KB
references/plugin-gates-and-fix-rounds.mdexecutablescript8.2 KB
references/production-reachability-review.mdexecutablescript5.0 KB
references/reconciliation-review-findings.mdexecutablescript3.1 KB
references/rest-api-review-checklist.mdexecutablescript13.0 KB
references/review-findings-adjudication.mdexecutablescript34.4 KB
references/scope-test-in-process-execution.mdexecutablescript4.2 KB
references/security-property-tests.mdexecutablescript14.2 KB
references/sqlite-migration-pitfalls.mdexecutablescript4.1 KB
references/sqlite-wal-review-findings.mdexecutablescript5.0 KB
references/sync-pipeline-review-patterns.mdexecutablescript13.2 KB
references/tool-mechanics-pitfalls.mdexecutablescript44.1 KB
references/upgrade-path-test-pattern.mdexecutablescript3.0 KB

references/confirm-loop-lessons.md

View on GitHub
# Confirm-Loop Lessons

Lessons from past diff-only confirm loops. Moved in v0.2 from the old `codex-confirm-loop.md`. The loop RULES (scope, severity floor, budget, stop conditions, gate) live in `review-commit/references/confirm-round-protocol.md` and win wherever these differ; CLI mechanics live in `review-commit/references/reviewer-dispatch.md`.

## Loop
1. Builder finishes round → verify independently (full pytest, node --check, live HTTP smoke) → commit with round tag.
2. Dispatch Codex on DIFF-ONLY scope: `git diff prev..new > /tmp/x.diff`, prompt lists the exact prior findings and asks RESOLVED/NOT-RESOLVED per item + SHIP/DO-NOT-SHIP. Diff-only avoids Muse-style timeouts and keeps reads bounded. Ask the confirm round to also hunt for new ordering bugs introduced by the fixes (e.g. 'no new pending-append ordering bug') — a pure verify prompt confirms fixes and stops hunting; two SHIP verdicts came only after this was added. The same confirm pattern works on SPEC documents (not just diffs): list prior findings with tags, ask RESOLVED / RESOLVED-BUT <residual> / NOT-RESOLVED per item, plus a mandatory NEW-problem hunt ("contradictions the revision introduced between sections, untestable criteria, over-engineering — list as NEW-N with severity, or 'no new problems'"). Spec-fix rounds commonly create contradictions elsewhere in the document even when every finding resolves; the NEW hunt is what catches them.
3. For each NOT-RESOLVED claim: reproduce it yourself with a minimal in-memory probe BEFORE writing the fix brief. Codex claims are ~80% real; reproduction tells you severity (real-world vs edge-case) and gives the builder a failing case. Also probe the CLAIM'S SCOPE against production reality: a bug keyed to a header layout the live sheet does not have is real code but low real-world severity — state both halves in the brief (the name-in-column-A bug of a past session was real in code but could not trigger on the production sheet).
   - Verify the reviewer's claimed repro shape, not a paraphrase of it. A reviewer states the triggering state precisely (e.g. 'the daily page with a source marker was written before the entity page'); rebuilding a NEARBY state (marker absent, or crash simulated instead of partial-write residue) can pass while the stated case still fails. Construct the exact pre-state the reviewer described, then run the retry. If your paraphrase passes but the reviewer insists, re-read their repro clause line by line.
   - A negative repro only counts as a fix verification if the probe could have demonstrated the bug at all: seed it with the data the alleged bug consumes or copies (the file a copy-loop would leak, the record a retry would write). An R3 symlink-escape probe was judged FIXED while it never seeded the file the backup loop copies — nothing could leak, so 'no leak' was vacuous; with the seed present the leak reproduced. Zero-effect probe = no evidence, not a pass.
   - When the reviewer saves its probe artifacts (Codex writes probes.py plus per-round captured outputs under its scratch dir), READ the script before rebuilding the repro from its prose. A 'persists vs fixed' dispute between two probes usually resolves in one read — the script shows the exact pre-state (which pages seeded, which writes failed, whether processed.json was deleted) that prose compressed away, and a variant probe you invent can pick the variant that happens to pass.
   - Two reviewers running the same codebase surface DIFFERENT bug classes — treat their findings as complementary, not redundant. One reviewer hunts crash/partial-write and config-edge classes, the other content-diff/staleness and portability classes; a 'converged' round where both confirm all prior items can still carry one new MAJOR each. That is why R1 uses several reviewers; confirm rounds then go only to reviewers with open findings (v0.2 protocol — the old "never end on a single reviewer's confirm" rule kept every reviewer hunting every round and was a main cause of R10+ loops).
4. Next brief: one item per fix, tight scope, 'stop after N items' clause, required new tests named. For multi-mechanism bugs (e.g. an export window closed by BOTH a sync-in guard AND a write-order change), require BOTH halves in one brief — a one-sided patch narrows the window instead of closing it, and the next confirm round reopens it. Cap builder tokens per round.
5. With parallel reviewers, HOLD the fix round until EVERY reviewer's verdict lands, then merge their findings into ONE deduped fix list (the same defect reported two ways is one item; a severity dispute is resolved by your own reproducing probe) and fix once — patching between confirmations makes later reviewers judge stale code. Commit each fix round immediately after your own verification (full-suite run, smoke), then dispatch the next confirm round on that round's fix diff to the reviewers that still have open findings. Do not batch multiple fix rounds into one review — each round's diff is the reviewer's scope.
6. A fix round may be scoped to reviewer feedback items from the USER, not just reviewer findings — operator feedback on a live site (hide PII, relabel controls, change navigation) goes into a numbered brief exactly like a review finding, with per-item FIX/DEFER reporting, before redeploy.
7. For a live-deployed app, redeploy immediately after commit (service restart picks up static files; verify through the tunnel before telling the user). Rollback path: stop service + revert tunnel ingress (keep the pre-change config backup) + delete the DNS record.
8. Operator UX feedback on a mobile door-tool is a first-class requirements source: walk the UI yourself (read the render code, not just the API) before writing the brief — several 'obvious' items have non-obvious root causes (a view-switch 'jump' is native anchor scroll from window.location.hash; the fix is removing hash routing, not scrolling after the fact).
9. When a fix requires a DESIGN-DIRECTION decision (layout density, hierarchy, how much brand vs tool), do not pick for the user: build 2-3 throwaway HTML sketches of the direction at the target device geometry, screenshot each, deliver the images, and recommend one. A direction chosen from prose descriptions gets reversed after it ships; a sketch comparison gets one decisive pick. Fold the picked direction into the same fix brief as the mechanical findings.

## Rules
- The three-reviewer deployment gate (all three reviewer lanes SHIP before live deploy) is the user's standing instruction. Its rules — run once on the net diff after the loop converges — live in `review-commit/references/confirm-round-protocol.md`; the lane models in `orchestrator/references/lanes.md`.
- Reviewer A's effort override is mandatory: Codex default effort resolves to `none` — never dispatch a review without `-c model_reasoning_effort="high"` (see `lanes.md`).
- Track every dispatched reviewer to a read verdict. On 2026-09-24 a round-2 Muse log was cleaned up unread because its completion notification arrived after convergence — the parallel reviewer's findings never got a verdict recorded. Rule: keep a dispatch ledger (round, reviewer, log path, verdict-read Y/N); before cleanup, confirm every ledger row is read or explicitly marked superseded-by-later-round. A stale late notification should trigger a ledger check, not just 'already processed'.
- Reviewer convergence on NEW CODE takes multiple rounds even with repro-before-fix discipline: the 2026-09-24 deep-research gates review ran 6 rounds (36 findings → 16 → 5 → 3 → 2 → 0/SHIP). Each fix round fixes exactly the reproduced findings. (v0.2: confirm rounds now look at the FIX diff, not a fresh full-state diff, and MINOR/NIT no longer keep the loop open — most of that tail would have been deferred, not re-reviewed.)
- A fix round can RE-OPEN previously-RESOLVED findings via an adjacent mechanism, so each confirm brief must list every prior-round item, not only the last round's. A partial-write tombstone fixed for the single-entity case returned in the two-entity case (marker matching keyed per-source, not per-referent); a backup containment fix passed its ancestor-walk probe while the copy destinations themselves went unchecked. Class rule for state-marker fixes: gate completion on the marker's REFERENTS (every artifact the marker claims processed must exist); for containment fixes probe the copy destinations, not just the copy roots.
- Scorer/formula code attracts an escalation ladder: vacuous-true → partial-overlap evasion → magnitude-suffix evasion → shared-figure masking → figure-absent bypass. Fix the CLASS, not the instance: after the 2nd numeric-contradiction finding, redesign the cap rule (e.g. issubset direction) instead of patching the specific pair. Ask reviewers to hunt the specific class you just fixed.
- Diff-only reviewers need per-fix status context (FIXED/NEW-PROBLEM per prior finding) or they re-report accepted limitations. List accepted limitations explicitly as do-NOT-re-report. Key each confirm prompt to the reviewer's OWN finding IDs (H1/M2 style) with the fix per ID, and add the reviewer-specific verification ask: Codex reliably EXECUTES probes from a diff — name the exact probes to re-run ('30-cycle dir reports found: 10, no full materialization'); Opus reads deeply and reports a per-ID fix-map check table plus must-fix vs low-priority split; Muse re-probes its prior cases from the diff alone and finishes last — budget for it.
- Dual-reviewer confirms (Codex + Muse in parallel) take ~15-20 min at high effort on typical rounds, but a Muse confirm with a long prior-findings list + repro probes can run ~50 min — arm monitor timers and do not diagnose a hang before ~60 min; expect Muse to be the slower finisher by a wide margin.
- Muse confirm output shape: verdict block of per-item RESOLVED/NOT-RESOLVED lines with an ID map, then new issues (severity + file:line), then a reproduction-evidence section listing what it ran. Its NOT-RESOLVED list can include items it scoped out of its own original review — read the ID map to distinguish genuine regressions from self-declared scope-outs before writing the fix brief. Its new-MAJOR finds are usually real (write-once staleness reproduced first try) but verify with an in-memory probe anyway; its MINORs skew publication/portability (license naming, Windows-compat shellouts).
- Two reviewers on the same codebase surface DIFFERENT bug classes: one hunts crash/partial-write and config-edge classes, the other content-staleness and portability classes. A round where both confirm all prior items can still carry one new MAJOR each — so use both in R1 (who reviews later rounds: `confirm-round-protocol.md`). Verify each reviewer's repro against the EXACT pre-state they described, not a paraphrase (a nearby state can pass while the stated one still fails).
- Verify a review fix actually LANDED before dispatching the next confirm round. The patch tool can report success with an EMPTY diff body (no unified diff printed) and still have changed nothing on disk — after any patch to reviewed code, re-read the target region or re-run the regression test and check the reproduced score/case flipped. An empty patch output plus a score that did not move means the fix never applied, and the next round reviews stale code.
- Review convergence (v0.1 full-state diffs) followed an evasion-class cadence: each round surfaces one more input class the previous fix missed (this session: magnitude words → year superset → comma/model-number tokens → unit runs → occurrence-level year boundaries — 5 confirm rounds, findings 3→5→1→1→0). That is the same-class pattern: under the v0.2 protocol the second same-class finding stops instance patching and triggers one class-level redesign brief instead of more rounds. Write each confirm prompt with the explicit do-NOT-re-report accepted-limitations list so the reviewer hunts NEW classes instead of re-litigating design choices.
- Killing a respawning CLI process (codex exec workers): a single `kill <pid>` fails because the dispatcher respawns — loop os.kill over `pgrep -f` output via execute_code until pgrep returns empty, instead of hand-killing a moving PID list in terminal.
- Vacuous-true tests are a real hazard: a regression test that passes for the wrong reason enshrines a false green. A coverage variant of this: a test built on a shared fixture that ALREADY carries the triggering signal (the marker, the field, the flag) proves nothing about the argument-form under test — strip the signal from the fixture base so the path under test must supply it, and assert the stripped base alone stays silent. Reviewers catch this ('passes even if operations is never read'); fix it proactively. Verify each new test fails against the pre-fix code AND that it fails for the stated defect reason (read the assertion error, not just the FAILED line). Three silent vacuity modes to check: (1) the monkeypatch hooks a layer the module does not use (patching builtins open when the module reads via os.read or Path.read_bytes) — hook the module's actual I/O path in its own namespace (module.os.read, not builtins) and count hook invocations to prove interception, and place the hook inside the race window (a hook that fires before the first fstat proves nothing about a read-between-stat race); (2) a keyword-presence assertion matches text inside an UNRELATED message the same call can emit — assert on the structured field (res['reason']) or prove the keyword is unique to the expected message; (3) a concurrency test that never runs concurrent code — sequential 'writers' prove nothing about locking; (4) a test that is RED for the wrong reason — a stale fixture helper from an earlier kernel cell (old marker format, stray trailing newline) produces 'malformed'-class failures that look like the defect but are probe artifacts; when a new test fails in an UNEXPECTED way (wrong reason string, wrong line number), rebuild the fixture fresh in that cell and read the module's actual wire format (parser regex, begin/end helpers) before treating the failure as RED evidence.
- Mutation-test each fix round's new coverage: temporarily gut the fixed function (early return / raise), run the suite, confirm the round's NEW tests fail (note the count), restore, re-run green. A fix round that lands 209 new test lines yet keeps passing with the fix deleted wrote tests around the bug, not through it — the gut-run failure count is the coverage proof, and it takes under a minute.
- Fixing one direction of a race leaves the symmetric direction unreviewed: a lock race 'fixed' on the acquire path can still be racy on the release path (an unlock-before-unlink gap lets a waiter acquire the doomed inode while a third writer opens a fresh inode). Require the confirm reviewer to probe BOTH directions.
- For a timing-flaky race probe, prefer a deterministic BEHAVIORAL probe over a source-text assertion: for lock races, count max concurrent holders — 3+ staggered writers, widen the release window by hooking the module's unlink AND its flock LOCK_UN with sleeps (in the module's own namespace, not builtins), assert max holders == 1. Prove the probe discriminates before trusting it: run it against the pre-fix module (git show <pre-fix>:<module> into a scratch dir) and confirm a different count there (2 vs 1). A raw source-index assertion (rfind('unlink') < find('LOCK_UN')) is a last resort: it pins comment wording (a harmless comment reword breaks it) and passes any code that merely got the text order right — if you keep one, index code tokens only, never prose, and treat its failure against pre-fix code as discrimination evidence to verify.
- A test asserting the WRONG expectation (e.g. vague claim + keyword-rich snippet scoring high) is the agent's own bias — probe the semantics before pinning the expectation.
- DO-NOT-SHIP from a diff-only reviewer is not blocking by itself — severity-triage first. A bug that cannot trigger on the production header is a code defect, not a launch blocker; say so in the brief and to the user.
- Ship decision is the user's, but recommend with a deadline: when wedding/launch is days away, weigh accepted-known-issues vs another round, and say what the next round would buy (which open BLOCKING/MAJOR it targets) — not "one more for safety".
- Convergence signal: each reviewer round finds ≤1 narrower edge. Round budgets and stop conditions: `confirm-round-protocol.md` (hard budget per tier; MINOR-only residuals ship).
- Keep a per-round ledger in chat (commit hash, diff size, tests, verdict) — user pings 'status' and expects the current round state instantly.
- Pin test counts in briefs from `pytest --collect-only -q | tail`, never from reading the file — a hand count that includes class-level fixtures and helpers inflates the number, and a brief that says N while pytest collects M makes the builder stop-and-report on a discrepancy that was the orchestrator's error (the honest stop is correct behavior — reward it in the status, fix the brief).
- For an integration/adoption batch wiring already-shipped behavior, state in the brief which tests are RED-new-behavior and which wiring tests may legitimately start green — RED-first applies to new behavior, not to wiring of shipped code; demanding RED on wiring invites vacuous tests that mock the shipped code into failing.
- Check the brief's own factual claims against repo history before dispatching: a fix brief that asserts a baseline fact (license family, prior behavior, config default) from memory can be wrong, and a builder deviation that CORRECTS the brief is a verified improvement to accept — with the brief error owned in the status report — not a compliance failure to force back. Judge deviations against git history, not against the brief text.
- Arm one sleep-timer per active background proc; when a timer fires late, answer 'stale timer — superseded' and stay quiet. Never re-derive state from timers; use git log/proc status — including ELAPSED TIME: a chain of 5-min timers was read as '17 min in' when the worker had started 2 min earlier (timer count ≠ elapsed time); take process age from `ps -o lstart -p <pid>`, not from counting timer firings.
- When a dispatched reviewer dies on a provider usage limit (Codex exits mid-run with 'hit your usage limit, retry at HH:MM'), do not rewrite the prompt: harvest its partial log tail for unverified probe evidence (treat as leads to reproduce yourself, never as findings), note the reset time, and re-dispatch the IDENTICAL prompt after the reset. If the user says to assess instead of the reviewer ('assess yourself instead of codex'), perform the confirm yourself against the prior round's exact finding list — reproduce each unresolved item with a probe, give per-item RESOLVED verdicts, check for regressions the fixes could have introduced, and record in the status that the named reviewer was substituted by operator instruction.
- Confirm prompts should carry the builder's DECLARED INTERPRETATIONS: when a fix required a semantic choice the brief did not dictate (a cache exemption, where a scan boundary lands, what counts as the observation's end), list each interpretation in the confirm prompt and ask the reviewer to accept or reject it with evidence. Unlisted interpretation choices get re-litigated as 'new issues' or silently accepted without a verdict; listed ones get an explicit ruling you can cite on ship. When the shipped contract pins a status word or behavior that differs from the SPEC prose (spec says 'refused', executable contract pins 'error' for validation rejects) and the exit code and prior-round tests match, adjudicate for the executable contract, list it as an accepted interpretation in the confirm prompt, and queue it for the spec note — do not force the code back to the prose. Also list prior-round residuals as pre-existing (with the evidence that an identical probe on the pre-fix tree behaves the same) so the reviewer does not reclassify a pre-existing residual as a new regression — a residual unchanged by the fix is a recorded-accepted item, not a new finding.
- When the user redirects a review to you mid-loop ('assess yourself instead of <reviewer>'), perform the confirm yourself against the prior round's exact finding list: reproduce each unresolved item with a probe, give per-item RESOLVED verdicts, hunt for regressions the fixes could have introduced, and record WHO was substituted and by whose instruction in the status. Cancel any timer waiting on the dead reviewer's reset — do not re-dispatch the substituted reviewer later in the same loop.
- Repro scripts from earlier rounds are one-shot probes, not persistent state: rebuild each round's repro from scratch in a fresh mktemp -d (stale state from a previous probe makes a 'fixed' check pass for the wrong reason). When a repro creates paths relative to CWD (a malformed path config can yield a literal `{}` dir), `cd` to the probe dir first and clean it after — never run malformed-path probes from the repo root.
- In a persistent code-kernel session, probe helpers defined in an earlier cell leak into later cells: a stale helper (one that appends a stray newline, uses an old marker format) silently changes the fixture and the probe reports a phantom defect the green suite disproves. Redefine every probe helper fresh in the cell that uses it. When a probe result contradicts a passing suite, suspect stale kernel state or a probe-format bug first — read the module's actual wire format (its parser regex, its begin/end helpers) before hand-building fixture strings; guessing the syntax burns probe rounds on 'malformed' refusals that are probe bugs, not module defects.
- A burst of file deletions in a repro (rm -rf of probe dirs) can trip a security scanner flag mid-command; batch cleanup into one trailing rm of explicit paths, not mid-probe interleaved deletes.
- Stale fixtures: probe DBs under /tmp, kill uvicorn smoke servers after use (process_manage kill), clean __pycache__ before git add.
- Dispatch each reviewer from a working directory it can actually operate in: `codex exec` refuses to run outside a git repo ('Not inside a trusted directory', exit 1 in <2 s, no findings) — run it from the repo root and reference the diff file by absolute path. `claude -p` reads only files under the cwd's allowed dirs — run it from the directory HOLDING the diff (a /tmp diff reviewed from the repo cwd comes back 'cannot read the file, VERDICT: none yet'). Re-run from the right cwd; never harvest 'no findings' from a run that exited before reviewing — exit code <0/1 with only an environment error is a failed dispatch, not a clean review.
- Codex sandbox flag values are exactly `read-only`, `workspace-write`, `danger-full-access` — `workspace-read` is not a value and dies in <2s with `invalid value`. Read-only sandbox cannot run pytest (crash with 'No usable temporary directory'); workspace-write CAN hang the review session on DB-backed integration suites (the reviewer stalls inside a slow test and the session dies with agent_close exit -15 before any verdict is printed). For pipeline repos: dispatch read-only static review (explicitly 'do NOT run pytest') and run the suite yourself as orchestrator — partial mid-death findings are still worth harvesting from the session log but never constitute a round verdict.
- Static reviewer findings are still worth validating yourself in parallel while the redispatch runs: for each unconfirmed finding, grep the exact function (shim mappers, settings gates, endpoint constants) before the re-review — self-verified findings can be fed into the redispatch prompt as explicit targets ('a prior attempt suspected X — verify'), which converts a wasted-look round into a targeted confirmation round.
- A multi-round loop's fix briefs live as durable files in the repo's plans dir (one per round, named fix-round-N.md), not in chat only: the resumed builder reads 'Read the fix-round brief at <path> and execute it fully' and needs nothing else. Each brief: per-finding section (behavior contract, exact fix instruction, test name for the failing-first test), ground-rules block (suite green after each commit, one commit per finding, no push, lint clean), report-back format (per-finding commit hash + files + test names).
- On deploy, record the access credential location AND offer delivery, but do not paste tokens unasked into chat; regenerate on request (user asked for a memorable token — a 48-hex secret is fine for ops, wrong for typing on a phone at a door).
finishing-a-development-branchobra/superpowersUse when implementation is complete, all tests pass, and you need to decide how to integrate the workStars: 322kUpdated Claude CodeCodexVerifiedInstall
systematic-debuggingobra/superpowersUse when encountering any bug, test failure, or unexpected behavior, before proposing fixesStars: 322kUpdated Claude CodeCodexVerifiedInstall
test-driven-developmentobra/superpowersUse when implementing any feature or bugfix, before writing implementation codeStars: 322kUpdated Claude CodeCodexVerifiedInstall
skill-creatoranthropics/skillsCreate new skills, modify and improve existing skills, and measure skill performance. Use when users want to create a skill from scratch, edit, or optimize an existing skill, run evals to test a skill, benchmark skill performance with variance analysis, or optimize a skill's description for better triggering accuracy.Stars: 198kUpdated Claude CodeCodexVerifiedInstall