moonlight-lupin/agent-skills / plugins/dev-pipeline/skills/review-lessons
review-lessons
VerifiedLibrary of lessons from past build-review loops — not a stage. Use when a review or fix round hits a specific case: a reviewer BLOCKING that contradicts the brief, a test that passes but checks nothing, a feature dead in production, SQLite/WAL migrations, MCP tool surfaces, REST API authz, app ↔ external-source sync, Android/APK verification, fill-template extraction, CLI/git mechanics, reviewer outages. Load the one file the case needs; never load the whole library.
references/loop-pitfalls.md
View on GitHub# Loop Pitfalls (lessons)
Pitfalls and red flags collected from past build-review loops. Moved in v0.2 from the old `build-review-loop.md`. These are lessons, not loop rules: round budgets, severity floor and stop conditions live in `review-commit/references/confirm-round-protocol.md`, and that file wins wherever these differ.
1. **Don't trust the builder's self-report** — "all checks passed" from Cursor is a claim, not a fact. Run checks yourself.
2. **Findings increasing between rounds = regressions** — stop and investigate before continuing.
3. **Commit between rounds** — commit after each fix round so reviewers can diff against the prior state. Rule that keeps the commit clean: `git add -A` stages the WHOLE repo even when run from a subdirectory cwd, so review diffs, FIXES ledgers, and reviewer logs parked at the repo root get swept into the build commit and shipped to reviewers as part of the tree. Root-`.gitignore` the workflow artifacts BEFORE the first commit, run `git status --short` before every commit, and from a subdirectory add explicit paths instead of `-A`.
4. **Builder and reviewer must be different models** — same model reviewing its own work has blind spots. Cross-model diversity is the whole point.
5. **Stale mock payloads mask integration bugs** — when a test mocks a function's return shape, and the real function's return shape changes, the test still passes against the stale mock. This is invisible to all review rounds unless someone explicitly compares the mock to the real output. Always verify mock payload shapes match the real function's actual return at each integration boundary (Step 4 check #6). One B1-class bug (5 commits, 2 review rounds, silent agent failure) was caused by exactly this.
6. **The orchestrator doesn't hand-code non-trivial fixes** — when the review loop is the active workflow, Hermes dispatches code changes through the builder lane. (v0.2: trivial fixes — ≤ ~10 lines, mechanical, no design choice — the orchestrator may make directly and must say so; see `review-commit/references/workflow-steps.md`. Routing every one-liner through a builder dispatch plus a full review round was one cause of long loops.) The original incident: The user corrected this mid-build: the orchestrator read the review findings and started patching a report module directly. The user said "didnt use cursor cli?" — Hermes had to stop, write a brief for the remaining fixes, and dispatch Cursor. The rationale: (a) the user chose this workflow specifically for cross-model diversity, so bypassing it defeats the purpose; (b) even small fixes can introduce regressions that only the reviewer catches; (c) dispatching through Cursor creates a commit boundary that Codex can diff against. For a trivial fix, make it and say so — never silently.
7. **Audit existing API surface before implementing "full CRUD" from a roadmap** — roadmaps describe desired state, not current state. Before dispatching a builder to "add CRUD endpoints", grep the existing routers and engine functions. One build assumed missing CRUD for all priority entities, but the REST API already had GET/POST/action endpoints. The actual gap was only PATCH (update) for draft records. Always grep the existing routers (`@router.` decorators) and the engine functions of every module before writing the brief.
8. **Cursor CLI is the BUILDER, Codex is the REVIEWER — never swap roles** — when the user says "have cursor and Claude code review", the reviewers are the reviewer lanes (Codex, Claude Code). Cursor CLI is the builder/fixer lane (models: `orchestrator/references/lanes.md`). The user corrected this explicitly in Aug 2026: "Reviewed should be codex not cursor" when I dispatched Cursor as a reviewer. The cross-model diversity principle requires the reviewer to be a DIFFERENT model than the builder. If Cursor built it, Cursor cannot review it — same-model review shares blind spots. The default reviewer pair is Codex + Opus. Cursor is only used for fixes after the review identifies findings. If the user says "review", dispatch Codex and/or Claude Code. If the user says "fix", dispatch Cursor.
9. **Round-2 reviews audit the previous round's FIXES — expect new findings in freshly changed code, and verify them before re-fixing** — when an external review returns after a fix round (e.g. "10 Fixed / 3 New"), the New findings are almost always regressions introduced by the fixes themselves. One round-2 review found exactly three: (a) the CSV formula-injection guard was lifted into a shared helper and applied to every exporter, but its prefix list contained `-`, so every negative number in six export surfaces (including a bank-upload CSV) exported as text `'-700` — a guard written for mostly-text list reports was inherited unchanged by financial exports where negatives are half the data; (b) a totals rewrite changed `total_debits`/`total_credits` from gross column sums to net-by-account while the line columns stayed gross, and the same commit patched the two DISPLAY surfaces (web template + CSV exporter) to sum the lines themselves — resolving the display but splitting the answer: JSON API and MCP returned one foot, UI and CSV another; (c) validation added at read time without adding the matching check at creation — the CLI could still issue names the API refuses to serve. Two structural lessons: **a value computed on one surface and re-derived on others will drift** — compute it once at the source and make every surface read the same field (the review's direction: the total should be computed once, in the shared engine function, not independently in the template and the CSV exporter); **validation added at one gate must be checked against every other gate for the same object** — validate at creation, not just at read. Also note the review's meta-observation: all three new findings lived in code that HAD tests added in that round — the tests simply lacked a negative number, a two-sided account, and an uppercase entity name. When fixing, add the missing assertion cases, not just the fix. A fourth regression mechanism from a later delta re-review: **a bulk copy step added to satisfy one finding can silently undo a validation fix from another** — a 'copy top-level files' change copied the source's own config files over the validated, credential-filtered manifests the same round had generated, reopening a BLOCKING the review had already closed. When a fix round adds any bulk copy or generation step, exclude tool-owned output filenames from the copy loop and generate validated outputs AFTER any bulk copy, never before.
10. **Multi-candidate external reviews: get scope decision before dispatching** — when an external review returns multiple refactor candidates (e.g. "4 deepening candidates, top recommendation X"), do NOT start building the recommended one immediately. Summarise each candidate with verified evidence, give your read, then ask the user which scope to run (top-only / top+paired / all-in-sequence / discuss). A user may say "all four in sequence" — which turns one batch into a 5-batch cycle with a different commit budget. Scope is the user's call; sequence and batching are yours. Validated on a 4-candidate deepening review; the user chose all four in sequence.
11. **Review findings that contradict the brief's design assumption — verify in source, then re-derive the fix rule, before dispatching** — when a reviewer flags a design decision the orchestrator itself encoded (Opus's B1: "v2 skip is wrong"), do not reflexively forward the finding to the builder. First verify the claim against source (here: `_ensure_hash_backfill` at db.py:487 runs before `_ensure_dr_cr_repair` at 488 — confirmed), then re-derive the correct rule (historical v2 digests attested repaired values → repair amounts + keep description converges with the chain), THEN write the red tests encoding the corrected rule and only then dispatch. The corrected insight went into the brief as the fix's central rationale — a builder given only "make the review finding go away" might have implemented a hash-rewrite (breaking tamper-evidence) instead of the value-convergence fix. Also: split review findings into fix-now (verified, cheap, testable) vs defer-with-record (need product decisions — rename atomicity, MCP scoping), and write the deferrals into the ADR's "Known deferred items" section in the same brief so they don't evaporate.
12. **Deferred review findings need a written destination in the same brief** — when a dual review returns more findings than can be fixed in one round (Opus: 8 MAJOR), do not silently drop the ones you skip. Split into fix-now vs defer-with-reason, and instruct the builder to write the deferrals into a durable in-repo document (e.g. an ADR section titled "Known deferred items") with one line each. This converts "we didn't get to it" into an auditable backlog the next session can pick up — and stops the NEXT review round from re-reporting the same items as new findings. Validate the builder's resolution table independently before committing: re-run the affected suites yourself and diff the contract-test file to confirm the builder didn't touch it. Validated on a deepening-cycle fix round 3: the builder's claimed test totals were independently reproduced before commit.
13. **A builder's "tests pass" mid-build is not completion — wait for the full-suite verification it was briefed to run** — when a long build (25-35 min) is briefed to run the full suite before reporting, intermediate probes (targeted suites passing at minute 25) are NOT the finish line. The builder may spend its remaining time on the full-suite verification + report writing. Wait for the process to exit (notify_on_complete fires); then verify. Do not commit based on mid-build probes, and do not kill a healthy long-running build to "check progress" — check progress with read-only probes (pytest on the contract-test file, git status) instead. Validated on a round-4 build: contract tests green at the 25-minute mark; the builder ran another 10 minutes of full-suite verification before exiting cleanly.
14. **Test fixtures must never reference real companies, clients, or the user's employer** — when writing contract tests or examples, use neutral fictional names ("Foo Holdings Ltd", "Acme"), never names connected to the user's actual world. The user corrected this mid-build: a review artifact used the user's real employer name, and the user interrupted the build to have every occurrence removed. Real names in fixtures leak business relationships into pushed repos, search indexes, and AI training data. Before committing any test file, scan fixtures for names that match the user's employers, clients, or domains mentioned in conversation/memory; replace with neutral fictions. Save durable "never use X" facts to memory AND treat every future fixture as suspect until checked.
15. **The reviewer commits — the implementer never commits** — the commit belongs to the party that verified the work. Builders (Cursor) run with commit capability disabled; reviewers (Codex/Opus) run read-only on the diff. A builder that lands a commit removes the review boundary — the reviewer can no longer diff the work against a clean base. Related discipline (ported from delegate-skills): between dispatch and the orchestrator's commit, the uncommitted working tree is the implementer's work — the only copy. Never run `git checkout`/`reset`/`clean`/branch switch before inspecting; `git status` + `git diff` + `git diff --cached`, open untracked files directly (no diff shows them), and only then decide — including a legitimate discard. Validated Sep 2026 on the delegate-skills port.
16. **A running builder reads its brief once at dispatch — superseding the brief in-place is a race, not a redirect** — the builder reads the brief file at startup; overwriting it mid-flight only redirects the builder if it happens to re-read later. A round where this gamble lost delivered the previous round's scope entirely, and the remaining findings needed a second dispatch anyway. Rule: never rely on in-place brief supersede to change a dispatched builder's scope. Let the round finish, verify what it actually did, then write a fresh numbered brief covering ONLY the remaining items with explicit context ("the previous round covered X — do not redo it"). After any ambiguous round, grep the builder's log for which brief it read before judging its output, and probe a canary behavior that distinguishes old-brief from new-brief work (e.g. the headline finding still reproducing = stale brief).
**Mid-build user steers are scope changes, not hand-edit invitations.** When the user sends a correction or addition while a build runs, do not edit the built files directly to absorb it — that bypasses the delegation contract AND leaves the brief stale, so the reviewer later flags a brief/code conflict that costs a round. Queue the steer: let the round finish, verify, add the steer to the fix brief (or a fresh numbered brief) with its acceptance criteria, and update BRIEF.md to match the new scope in the same commit. The brief, the code, and the review must agree on scope before the next dispatch.
17. **Re-adjudicate reviewer findings against current HEAD, probing through the public entry point** — a reviewer's report is a snapshot of an earlier commit; if fix commits landed after the reviewer started, its "regression" claims may describe code already fixed. Re-run the reviewer's exact probe against HEAD before writing a fix brief. Probe through the PUBLIC entry point (the CLI command or top-level function), never by calling the guard helper in isolation: an isolated helper call can show a leak the guarded call path already prevents, and a single guarded-path probe can miss a leak on a sibling call path for the same data class (e.g. env vs headers in the same server converter) — check every call site of the guard before declaring an item closed.
18. **Verify the verification harness before reporting a fix as failed** — when an independent re-probe says a fix did NOT land, first rule out harness bugs: a probe can compare the wrong shape (parsed dict vs field: `safe_load('description: "no"')` returns a dict — compare `.get('description')`, not the dict), count intended behaviour as an anomaly (a tokenised constant after a fingerprint fix), or probe an isolated helper instead of the public path (see #36). The agent-skills #12/#13 loop (Sep 2026) produced a 0/9 yaml-roundtrip reading and a 500/500 churn reading that were BOTH orchestrator harness bugs; the builder's fixes were correct. Rule: before relaying "the fix didn't land", re-derive what the probe actually asserts and rerun with a corrected probe; never adjudicate a builder's fix off a single suspicious probe. Also rule out missing probe preconditions: a reproduction probe that skips setup the real path requires (registered capability facts, seeded state rows) makes correct code look broken — derive the probe's fixture from the passing test's setup, not a minimal guess. Validated on an orchestrator verification round: a degraded-skip probe without registered facts showed no skip while the dedicated test passed.
19. **Reproduce every reviewer finding before it enters a fix brief** — run the reviewer's exact probe (or a minimal equivalent) against current HEAD before dispatching. Three payoffs, all observed in the #12/#13 loop (Sep 2026): (a) wrong claims get caught BEFORE costing a builder round — the hardlink data-loss claim failed to reproduce twice under the orchestrator's probes and was twice re-verified by the builder before the real failure mode surfaced; (b) builders get exact probe scripts instead of prose, so they verify with the same evidence; (c) each reproduced finding becomes a permanent regression test in the same round. A finding that cannot be reproduced gets marked "verify-first" in the brief with the probe evidence attached — never silently accepted, never silently dropped.
20. **Assume fixes introduce findings — probe adjacent logic on every fix round** — from round 3 onward in the #12/#13 loop, roughly half of all findings were regressions caused by EARLIER fixes: tighter arg validation added → validation error text echoed the raw secret into the report (G1); row-collapse fix added → unique nested content destroyed (D3); constant-promotion fix added → over-fired on repeating instances (E1); schema-validation fix added → bulk copy step overwrote the validated manifests (F1/R1). Rule: every fix brief requires (a) a regression test for the fixed finding AND (b) a probe of the immediately adjacent behaviors — the same guard's sibling call sites, the same rule's boundary values, anything sharing the mutated function. The convergence exception (deep sweeps) covers reviewers finding NEW code paths; this pitfall covers YOUR fixes breaking boundaries they touch. Both are normal; neither means the loop is failing.
21. **Verify two-sided fixes on BOTH sides** — a fix round can satisfy the probe on the surface you test and leave the sibling surface broken: a divergence disclosure was rendered in text but NOT in HTML because only the text-renderer branch was updated. When a finding spans multiple output surfaces (text/html/markdown) or both persistence paths (module function vs method), probe each surface separately in the same verification pass. Single-surface verification declared a round verified while the HTML path stayed silent until the next review round caught it.
22. **Mid-build user requirement additions go into the NEXT numbered brief — never hand-edited into the running build; the spec update lands in the SAME commit round** — when the user adds or changes requirements while a build is in flight: (1) capture the addition in SPEC.md immediately with NEW numbered subsections (e.g. §3b, §6b) rather than rewriting existing sections — the in-flight round read the old spec, and numbered additions make the delta unambiguous to the next round and the reviewer; (2) update endpoint lists, data model, and UI sections in the same spec edit so it stays internally consistent (a column-map change must propagate to the API contract and data model in the same edit — an incomplete spec edit costs a review round); (3) let the in-flight round finish on its original brief; (4) write a FRESH numbered brief (BRIEF2.md — never overwrite the brief a dispatched builder already read, see 'A running builder reads its brief once at dispatch') naming the prior commit as the baseline ("extend it, do not rewrite from scratch") and enumerating ONLY the delta. Never let a user addition silently ride along un-specified — an undocumented feature is a review finding waiting to happen.
23. **UI-restructure rounds need a fresh-session browser smoke by the orchestrator — builders cannot run a live browser** — a filter/layout change can pass every pytest HTML assertion and JS DOM-stub while breaking the real first-load path: the default-selection state (e.g. "no category selected" renders an empty list on a fresh session) is exactly the state stubs never exercise. Builders routinely report "could not click through a live browser in this session" — treat that as a verification gap the ORCHESTRATOR owns, not a residual risk to mention in the delivery. After deploy, drive the real URL headless as a FRESH visitor (login via in-page fetch, reload, count rendered list items, read the control's value) before committing or calling the round shipped. Also make "empty list" vs "no search matches" distinguishable states for staff — they look identical to a filtered-list UI but mean different things.
44. **Read back long spec/brief writes before attesting and dispatching** — a long file write can land garbled or truncated (mid-write corruption); the write tool's 'verified' flag confirms bytes on disk, not that the CONTENT is coherent. A builder reads the brief once at startup: a garbled spec silently scopes the entire round wrong, and `attest_file.py attest` on the garbled file locks the corruption in as the tamper-evidence baseline. Rule: after writing any SPEC.md/BRIEF.md you intend to dispatch, verify the body reads clean end-to-end (spot-check head, middle, and tail via read_file or the write result) and re-write on any garble — BEFORE attesting, committing, or dispatching. Never attest a file you have not read since the last write.
25. **Derive spec data shapes from the real producers, not from your mental model — a wrong-shape spec builds dead code that its own tests pass** — when a feature snapshots or aggregates data produced elsewhere, read the producer's actual return statements and copy their literal shapes into SPEC.md: field names, flat vs nested, numeric vs string (one briefing collector returned `"outstanding_ar": "0"` as a STRING — the trend flattener silently dropped it everywhere). A spec written from mental model produces three failures at once: aggregation logic that never fires on real payloads, counters probing field names that exist nowhere in the repo, and contract tests built from the same wrong spec that pass against fixtures. The builder implements the spec literally and will not flag the mismatch — that is correct builder behavior, so the obligation is on the spec author. Before writing SPEC.md, grep each producer's return statement and one live record; if a mismatch surfaces at review, reconcile with a probe of a real payload and fix at the producer or the spec, not with a silent coercion in the middle. Validated on a mid-build review round (reviewer finding: "the spec described shapes the codebase doesn't produce; the builder implemented the spec literally").
26. **Multi-reviewer gate: hold the fix round until EVERY confirmation verdict lands, and demand EXECUTED probes in confirm-round briefs** — dispatching fixes while one reviewer still runs forces an extra round for its late findings; keep a per-reviewer fix map and send each reviewer only the findings it owns (they have no memory of other reviewers' rounds). In confirm-round briefs, ask executable reviewers (Codex, Muse) to RECONSTRUCT the changed function from the diff and RUN it against in-memory fixtures with numbers in the answer ("consumed N entries", "len == CAP") — one executed probe caught a bounds leak (52 consumed under a 50 cap) that two consecutive rounds of static reading had both passed over. Reading a diff verifies intent; executing it verifies the bound.
27. **Verify a dispatch exists before reporting it — an announced-but-never-executed dispatch surfaces only at status time** — "dispatching the builder now" in a delivery message is not a dispatch; if the turn ends without the tool call, the user's next "status?" finds no process and the workflow has silently stalled. Before any "builder running" or "builder dispatched" line, confirm the process exists (pgrep the CLI command or check the background-process list). Pair with tool-mechanics #3: a `--print` dispatch missing `--trust` exits 0 immediately, so "completed normally" on a suspiciously fast build means it never ran.
28. **Assert resource/tokens bounds EXACTLY by instrumenting the yielding API — behavior tests with modest fixtures cannot distinguish a leaking bound** — a cap enforced with `sorted(iterable)` (materializes everything), or a limit checked AFTER consuming entry N+1, or checked before a `continue` that skips non-matching entries, all pass every functional test whose fixture stays under the cap (30 dirs, 30 tests) and fail only at real scale. Test pattern: monkeypatch the yielding API (`pathlib.Path.glob`) with a counting generator, build a fixture LARGER than the bound (60+ entries, no matching items), and assert `consumed <= BOUND` — this catches materialization, off-by-one, and per-pattern-budget-reset (each pattern restart must share one counter) in one assertion. Companion arithmetic rules: truncation text must slice to cap-minus-one BEFORE appending the ellipsis marker (slice-to-cap + marker = cap+1); the exact-cap assertion needs a non-whitespace cut point (`rstrip()` inside whitespace shifts the length and can mask the off-by-one); a numeric-0/bool guard must reject `bool` explicitly (`True` is an `int` subclass and passes `> 0`).
## Red Flags
Observable symptoms, each mapping to the rule that fixes it. The rule's single source of truth
lives in **## Pitfalls** (loop discipline), **## Workflow** (steps), or the **Verification Checklist** —
do not add symptom-only duplicates of those here. The items below are rules that exist nowhere else:
- BLOCKING issues ignored because "loops exhausted" — deliver REQUEST CHANGES, not silent approval. The loop budget bounds rework, not honesty.
- Missing role gates on write endpoints — `require_auth` is not `require_write_role`. Authentication without authorization is an open endpoint.
- A value computed on one surface and re-derived on others — totals summed in template + exporter while the DB holds another number. Compute once, propagate the result.
- A guard/sanitizer applied unconditionally to a field whose domain includes values that look like the threat — over-matching guards corrupt legitimate data. Parse and match the actual pattern.
- Validation added at one gate (read/resolve) without checking every other gate (creation/registration) for the same invariant.
- A refusal/degradation error that states the cause but not the remedy — name the flag or action that fixes it.
- Multi-step state writes (register + set label) where the idempotent re-run cannot repair a mid-sequence failure — make each step independently idempotent.
- Build artifacts (BRIEF.md, review diffs, review outputs) committed to the repo root — these are internal scratch, keep them out of the repo (see Release Hygiene Checklist).
## Domain checklist items (from the old verification checklist)
Apply when the change touches these areas:
- [ ] Cross-surface property tests (isolation, scope, role) written when adding a new surface or before adding a new vertical module — see `references/security-property-tests.md`
- [ ] Property tests actually execute — assert minimum execution count; use in-process registry calls, not subprocess with name transforms; catch `SystemExit` separately from `Exception`
- [ ] Exempted routes in property tests have a real separate test covering the same property — not just a comment claiming "tested separately"
- [ ] Argument-dispatching handlers split into per-subcommand handlers with honest scope classification — no handler whose scope depends on a runtime argument
- [ ] DB-level triggers enforce accounting invariants (factor sums, balance constraints) — not just application-level checks
- [ ] Polymorphic link tables use TEXT for target_id — supports both integer IDs and string account codes
- [ ] Cross-company polymorphic links blocked — target_id == company_id for company targets
- [ ] SQLite table-rebuild migrations preserve dependent rows — rename dependent tables aside before DROP+rebuild, never rely on PRAGMA foreign_keys=OFF inside a transaction
- [ ] Bulk operations are atomic — pre-validate all items, single-transaction insert, no per-item commits
- [ ] Base-currency exchange rates forced to 1 — reject non-unit rates when transaction_currency == reporting_currency
- [ ] Opus 529 handled with `--fallback-model sonnet` retry — review completes even when Opus is overloaded
- [ ] Cursor API resource_exhausted handled — either retry later or Hermes builds directly with contract tests
- [ ] Dual review findings synthesized into cross-model overlap table — both reviewers' findings addressed
- [ ] Tests validating old mechanisms updated to validate new mechanisms after architecture changes
- [ ] Local installation updated after push (stash, pull, resolve, verify)
- [ ] No build artifacts committed to repo root (BRIEF.md, review diffs, review outputs)
- [ ] Idempotent/registration paths suppress duplicates in SQL (`ON CONFLICT ... DO NOTHING`), not by catching IntegrityError — PG aborts the transaction (see Red Flags — idempotent multi-step writes)
- [ ] "Exists" fast paths verify the existing object matches the request; mismatch returns an error naming the occupied resource (see Red Flags)
- [ ] Refusal errors name the remedy (flag/command), not just the cause (see Red Flags)
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| finishing-a-development-branchobra/superpowersUse when implementation is complete, all tests pass, and you need to decide how to integrate the work | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |
| systematic-debuggingobra/superpowersUse when encountering any bug, test failure, or unexpected behavior, before proposing fixes | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |
| test-driven-developmentobra/superpowersUse when implementing any feature or bugfix, before writing implementation code | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |
| skill-creatoranthropics/skillsCreate new skills, modify and improve existing skills, and measure skill performance. Use when users want to create a skill from scratch, edit, or optimize an existing skill, run evals to test a skill, benchmark skill performance with variance analysis, or optimize a skill's description for better triggering accuracy. | Stars: 198k | Updated | Claude CodeCodex | Verified | Install |