moonlight-lupin/agent-skills / plugins/dev-pipeline/skills/review-lessons
review-lessons
VerifiedLibrary of lessons from past build-review loops — not a stage. Use when a review or fix round hits a specific case: a reviewer BLOCKING that contradicts the brief, a test that passes but checks nothing, a feature dead in production, SQLite/WAL migrations, MCP tool surfaces, REST API authz, app ↔ external-source sync, Android/APK verification, fill-template extraction, CLI/git mechanics, reviewer outages. Load the one file the case needs; never load the whole library.
references/tool-mechanics-pitfalls.md
View on GitHub# Tool Mechanics Pitfalls
Reached from SKILL.md ## Pitfalls. CLI flags, polling behavior, output handling, and
environment quirks for the builder/reviewer CLIs (Cursor, Codex, Claude Code). Re-read when
a loop involves dispatching or monitoring those CLIs.
## Contents
- Items 1-49 — native CLI flags, polling, timeouts, output handling, environment quirks
- Items 50-61 — in-loop mechanics split from SKILL.md pitfalls (2026-10-16 refactor)
1. *Codex needs a git repo** — always `git init && git add -A && git commit` before dispatching Codex review.
2. *Codex `--sandbox read-only` can't run tests** — use `--sandbox workspace-write` if tests need to run, or `--sandbox danger-full-access` for `~/.hermes/` paths.
3. *Cursor `--trust` required for headless** — always pass `--trust` (or `--force` which implies it). The missing-`--trust` failure is SILENT: `agent --print` exits 0 with a 'Workspace Trust Required' banner, and the completion notification reads 'completed normally (exit code 0)'. An instant completion on a build dispatch means the build never ran — check the log for the banner and re-dispatch with `--trust` before reporting any status.
4. *Cursor `--force` required for changes** — `-p` alone only proposes, never applies.
5. *Codex output may be truncated** — use `process(action="log", limit=800)` for full output.
6. *Cursor output may be empty in poll** — use `process(action="log", limit=800)` for full output. The `tail -5` in the command only captures the last 5 lines for the preview.
7. *NEVER pipe review/agent output through `tail` — write to a file instead** — when dispatching Cursor or Codex for a review (especially Opus), do NOT pipe the command through `tail -5` or `tail -800`. The `process(action="log")` call only sees what the pipe already truncated — the full review text is lost before it reaches the process log. Instead, instruct the agent to **write its review to a file** (`/tmp/review-output.md`) and then `read_file` that file. This is the only reliable way to capture full multi-page review output. The `tail` pipe truncates at the shell level before Hermes ever sees it.
8. *Write the review prompt to a file** — don't inline long prompts. Codex reads files better than parsing huge inline prompts.
9. *Verify CI's exact installation extras in a clean venv** — local editable environments often already contain optional dependencies. If application imports an optional adapter (for example FastMCP) during test/API setup, reproduce the workflow's `pip install -e ".[... ]"` in a disposable matching-Python venv and run its exact test command before pushing. Update CI extras deliberately rather than assuming a local pass proves the dependency contract.
10. *Codex full-suite timeout under sandbox** — Codex's bwrap sandbox adds overhead that can push a 3.5-min test suite past its per-command timeout. When Codex times out mid-suite, it falls back to running targeted test files. This is sufficient for review (Codex verifies the changed files pass). Do NOT interpret a Codex suite timeout as a test failure. If you need Codex to confirm the full suite, tell it to run specific test files in the review prompt instead of the whole suite.
11. *The `todo` tool requires a list of objects, not a JSON string** — pass `todos=[{"id": "t1", "content": "...", "status": "pending"}]` as the actual parameter, NOT a JSON-encoded string. Passing a string produces `{"error": "todos must be a list of objects, got unparseable string"}` repeatedly. This is a parameter type mismatch, not a formatting issue.
12. *User check-in cadence: poll ONCE per cycle, not continuously** — when the user says "check in every 5 mins", dispatch the build with `notify_on_complete=true`, then poll ONCE after ~5 minutes to report status. Do NOT loop-poll between check-ins. Each poll burns a tool iteration. Use a cron job (every 5 min) for recurring status updates if the user wants automated check-ins. Between check-ins, do productive work: write the next brief, plan the next batch, or update docs. See `review-commit/references/background-build-polling.md`.
13. *Codex NIT/MINOR fixes: Hermes may apply directly** — when Codex returns only NIT or MINOR findings (stale comments, docstring additions, test assertion tightening), Hermes MAY apply those fixes directly without dispatching Cursor. The "orchestrator never hand-codes" rule ('Cursor CLI is the BUILDER, Codex is the REVIEWER — never swap roles') applies to code logic, not to documentation or test-quality tweaks. If the fix changes behavior → dispatch Cursor. If it's a comment, docstring, or test assertion → apply directly. MAJOR findings in test-only code (test fixtures, test assertions, test setup) are also safe for Hermes to fix directly since they don't change production behavior — the 5.16 Batch 1+2 reviews had Hermes fix MAJORs in test files (account target_id TEXT vs INTEGER, cross-company test setup, fixture env var ordering) without dispatching Cursor. MAJOR findings in production code (engine logic, API handlers, SQL migrations) must go through Cursor.
14. *Opus 529 (overloaded) — retry with Sonnet fallback** — Claude Code Opus dispatches can fail with `API Error: 529 Overloaded`. This is a server-side issue. Retry the dispatch with `--fallback-model sonnet` (or wait and retry Opus). The 529 is transient. Do not treat it as a permanent Opus failure. The review will still complete with Sonnet, though quality may be lower. If the user specifically requires Opus, retry after a short wait.
15. *Cursor API resource_exhausted — drop to medium tier before falling back to Hermes** — when `cursor-grok-4.6-high` returns `RetriableError: [resource_exhausted]` after 3 connection-lost retries, do NOT retry the same tier. The error is a gRPC server-side capacity rejection, not a personal quota limit. Drop to `cursor-grok-4.6-medium` and re-dispatch. Medium tier succeeded on a 12-finding fix build (19 files, 397+/193-) where high tier failed 4 times in the same session (Aug 2026). Check `https://status.cursor.com/` — when CLI shows "Degraded Performance," high tier is affected first; medium has more capacity headroom. If medium also fails, fall back to `composer-2.5` (lightest internal model). Only if all internal tiers fail: Hermes builds directly with contract tests (write TDD red, implement, verify, run full suite, commit). The user accepted Hermes-built code when all Cursor tiers were unavailable. The cross-model review can happen later when the API recovers.
16. *Claude Code `--fallback-model` flag for Opus retries** — when re-dispatching Claude Code after a 529, add `--fallback-model sonnet` to the command. This tells Claude Code to fall back to Sonnet if Opus is still overloaded. Without this flag, the retry will fail again with the same 529.
17. *Set up a monitoring cron job when the user asks for periodic check-ins** — when the user says "monitor every 5 mins" or "check in every 10 mins" while a background build or fix is running, create a cron job (`cronjob action='create'`) with the schedule set to the requested interval, a repeat count of ~12 (covers ~1 hour), and `enabled_toolsets=['terminal']` to minimize token overhead. The cron prompt should poll the specific `session_id` ONCE and report status in 2-3 lines. This is cheaper and more reliable than tight-polling from the main conversation, and it lets the orchestrator do productive work between check-ins. Remove the cron job (`cronjob action='remove'`) when the build completes or the user says to stop. The cron job delivers status updates back to the origin chat automatically.
18. *Claude Code print mode can't write files as root** — `claude --max-turns N -p "write review to file.md"` fails with "write permission denied" in print mode, and `--dangerously-skip-permissions` is rejected when running as root/sudo ("cannot be used with root/sudo privileges for security reasons"). The workaround: redirect stdout to a file (`claude -p "output your review as text" > review.txt 2>&1`) and tell the agent to output the review directly as text instead of writing to a file. This captures the full review without any file-write permissions. Validated on a review dispatch (3 failed attempts before this workaround succeeded).
19. *Claude Code max-turns 50 is too low for large reviews** — a review of an 82-file diff (12K+ lines) exhausted all 50 turns just reading files, never producing output. Use `--max-turns 100` for large reviews. Also tell the agent to be efficient: "read only the key files" and "output directly as text" rather than reading every file.
20. *Claude Code `--dangerously-skip-permissions` rejected as root — use Cursor CLI with Opus model instead** — when dispatching Claude Code (Opus) for review, `claude --dangerously-skip-permissions -p "..."` fails with "cannot be used with root/sudo privileges for security reasons" when running as root (which Hermes does). The fix: dispatch Opus via Cursor CLI instead: `agent -p --force --model "claude-opus-5-5-high" --sandbox disabled "..."`. This routes the Opus model through Cursor's infrastructure, bypassing Claude Code's root restriction. The review quality is the same — the model is the same, only the CLI wrapper differs. Validated on a review dispatch (Claude Code failed exit 1, Cursor CLI with Opus succeeded and produced a thorough 8-BLOCKING review with executable probe scripts). Model ID updated 2026-09-23 to the current Opus generation (Opus 5.5). Also works for other Claude models via Cursor CLI if Opus is overloaded.
21. *Update local installation after push — stash, pull, resolve, verify** — after pushing to GitHub, update the local plugin installation (`~/.hermes/plugins/<plugin>/`). The local copy often lags behind and may have uncommitted local customizations. Pattern: (1) `git stash push -m "local-changes-before-upgrade"` to save local changes, (2) `git fetch origin main && git merge origin/main` to pull upstream, (3) `git stash pop` to reapply local changes, (4) resolve any conflicts (prefer upstream for files that were updated upstream; keep local for truly local customizations), (5) `git checkout origin/main -- <file>` to force upstream version when local changes are stale, (6) run the test suite to verify. Watch for untracked files that block merges (`rm` them before merging). Validated on a local plugin update (stale local customizations replaced by upstream versions).
22. *Claude Code (Opus reviewer) sandbox cannot read /tmp — put the review prompt in the repo or inline it** — `claude -p` running from a project directory is filesystem-sandboxed to that directory; it cannot read `/tmp/review-prompt.md` (permission prompt declined in print mode) and cannot run pytest (permission-gated). The review still completes from static analysis, but the reviewer must be TOLD the review focus inline. Pattern: give Claude Code the review focus list inline in the dispatch command (or copy the prompt file into the repo working dir), and accept static-analysis-only findings — Opus's static review still caught a BLOCKING the builder missed. Codex (`codex exec --sandbox read-only`) reads /tmp fine but cannot run tests (pitfall #15). Plan each reviewer's prompt placement around what its sandbox permits. Validated Aug 2026 on the deepening-cycle dual review.
23. *Set up ONE clean reference venv at session start; pin it in every builder brief** — the host interpreter (e.g. Hermes's own venv) often has incompatible transitive packages that make red contract tests fail for the WRONG reason (Beta Gate session: `ImportError: cannot import name 'request_ctx' from mcp.server.lowlevel.server` from a stale `mcp` package — nothing to do with the feature being tested). Fix: create one venv up front (`python3 -m venv /tmp/<proj>-venv --upgrade-deps && pip install -e '<repo>[extras]'`), then (a) use it for ALL orchestrator verification runs, (b) tell every builder brief "Use THAT interpreter, not system python" with the exact path. One venv also becomes the source of truth for release constraints (pip-freeze pattern: `pip freeze` from the tested venv = the tested dependency combination). Cross-check failure reasons only AFTER the environment is sane — otherwise you cannot distinguish harness failure from honest RED (see #47).
24. *Model tier per task class: safety-critical code → grok-4.6-high; docs/mechanical → composer-2.5** — documentation reconciliation, runbooks, and constraints-file generation need source-reading accuracy but not deep implementation reasoning. composer-2.5 completed the 9-file docs pass (~7 min) and the constraints+Dockerfile+guard-test build (~7 min) in parallel with a grok-4.6-high code build. Reserve grok-4.6-high for anything that changes runtime behavior. Reserves the high tier's capacity and shortens the critical path.
25. *Patching function-lazy imports: patch the real module, not `mod.requests`** — when the code under test does `import requests` INSIDE a function (requests is imported lazily inside that function), `patch("providers.mod.requests.post")` raises `AttributeError: module has no attribute 'requests'` because the module attribute doesn't exist at patch time. Harness failure, not honest RED. Fix: `patch("requests.post", side_effect=...)` — patch the real module that the function will resolve at call time. Symptom check: if ALL your new tests fail with AttributeError on an import/attribute chain, fix the harness first; only assertion failures count as honest RED (see see `pitfalls-fix-round-verification.md` — self-authored red tests two-bug triage). Validated on a first contract-test batch (5/5 initially failed for the wrong reason).
26. *Testing guardrailed write methods: set the auto-approve env var in tests** — provider methods wrapped in `@guarded(...)` (a workspace-guardrails decorator) refuse to run in tests with "requires confirmation but no TTY available" — the error message names the auto-approve env var. The guardrail block is a harness failure, not the behavior under test. Add an autouse fixture in the new test file that sets that env var via `monkeypatch.setenv(...)` — several existing suites already use this pattern; grep for `AUTO_APPROVE` in tests/ before assuming a refusal is real. Note the two meanings: in production the env var is a break-glass switch (never set it to bypass a user-facing block), but in the test suite it is the standard mechanism because there is no TTY to confirm with.
27. *rtk transparently filters terminal output — bypass it for review-critical commands** — the rtk-rewrite plugin mutates `terminal` commands before execution; `cat` rewrites to `rtk read` (signatures only, bodies stripped) and pytest/git filters collapse output. Load-bearing output (review text capture per pitfall #7, full diffs, gate checks) must run raw: keep such commands in `~/.config/rtk/config.toml` `[hooks] exclude_commands`, or prefix once with `RTK_DISABLED=1`. Spot-check `rtk rewrite "<cmd>"` (exit 3 = rewritten, 1 = passthrough) when a command's output must be complete — silent filtering looks like a clean result and hides the evidence a review needs.
28. *Verify against branches in a `git worktree`, never stash+checkout in a live tree** — `git worktree add /tmp/<name> <branch>` puts the comparison branch in a scratch dir, leaving the live tree (and any uncommitted builder work) untouched. `git stash` + `checkout` + `stash pop` in that tree risks a pop conflict on files unrelated to your question, leaving UU states that block every subsequent dispatch until resolved. Full details + the untracked-leftover-test pitfall: SKILL.md pitfall #33.
29. *Mid-build steering: background `agent` has no stdin — supersede the brief FILE* — `process(action="write")` fails with "Process stdin not available (non-local backend)" for Cursor CLI dispatches. The builder re-reads its brief by path at natural checkpoints, so rewrite the brief file in place (`cp NEW.md BRIEF.md`) and verify uptake by the builder's log (does it name the new items?) plus a canary probe (does the headline fix's behavior flip?). Items the builder never re-read must be re-probed after exit and dispatched as an enumerated follow-up. Full procedure: `review-commit/references/persistent-loop-plan.md` steering rule #6. Recovery for a stash-pop conflict this causes: `git checkout -q HEAD -- <conflicted files>` restores committed state (test-only edits, no builder work lost).
30. *Fixture plugins for contract tests: verify the harness against the REAL pipeline before committing the RED baseline** — when tests drive a two-phase CLI (analyze → convert), the test helper must run BOTH phases; a helper that invokes only the final phase fails with a confusing empty-stderr assertion failure that looks like the missing feature. Run the real pipeline once by hand first and capture the exact invocation chain. Same for spec-validation probes: a schema-validation probe missing the document's required `$schema` field rejects everything, making real findings look like false positives. Build the probe against known-good input FIRST. (Distilled from the agent-skills round; generalizes pitfall #18's harness-iteration rule to schema-validation probes.)
31. *A wrong orchestrator-written contract test blocks the builder — the stop is correct, fix the test the same turn** — when a contract test asserts an output path/shape that contradicts the actual base-branch behavior (e.g. asserting `plugin.yaml` at `<output>/` when the code writes `<output>/<name>/plugin.yaml`), the builder will correctly STOP with "blocked: untouchable test is wrong" rather than edit it. That stop is the desired behavior (see `review-findings-adjudication.md` — own the fixture fix). The fix is a one-line orchestrator test edit + immediate re-run + re-dispatch with "test fixed, resume" — do not route it through Cursor and do not leave it pending. The blocked item gates everything behind it.
32. *Contract tests must mirror the real pipeline's OUTPUT LAYOUT, not the obvious path** — before writing the RED test, run the base branch's real pipeline once and note where files actually land (e.g. `<output>/<name>/plugin.yaml`, not `<output>/plugin.yaml`), which artifacts are named, and what the report claims. A test written from the feature description alone asserts the wrong path and blocks the builder (pitfall #31's most common trigger). Same for multi-output tools: enumerate ALL outputs (manifest + reports + skill dirs) in the assertions, not just the headline file — silent extra/missing files are invisible to a single-path assert.
33. *Alias-aware byte comparison: capture reference bytes BEFORE the operation* — when a probe checks 'did operation X modify file A', copy A's bytes to a reference file BEFORE running X, then compare. If A can be hard-linked or otherwise aliased to the operation's destination (e.g. the extraction output path hard-linked to the source), reading either path after the operation shows the other's contents and a naive before/after read lies. Also check `os.stat().st_ino` / `os.path.samefile` when aliasing is possible. A hard-link overwrite claim was dismissed as 'not reproduced' because the probe compared against a misread; the builder's careful re-probe (reference bytes first, inode check) showed the source HAD been truncated through the shared inode.
34. *Claude Code reviewer sandbox: pass `--add-dir` for prompt files outside the repo, never `--dangerously-skip-permissions` as root* — `claude -p` is filesystem-sandboxed to the working directory plus declared add-dirs; a review prompt at `/tmp/...` fails with "permission not granted / only allowed working directories" and the dispatch burns a full run producing no review. Fix: `claude --add-dir /tmp/<briefdir> -p "..."` when prompts/diffs live outside the repo, or copy them into the repo (pitfall #22's options). Separately, `--dangerously-skip-permissions` is ALWAYS rejected when Hermes runs as root — never add it; stdout-redirect capture (`> review.txt`) plus text output works (see #18). Root restriction is permanent on this host, not transient.
35. *A builder resolving its own test-import bug via `builtins` is a scope violation — remove the hack, fix the test yourself* — when the attested test file has an authoring bug (imports name A, calls name B), a builder under "do not modify tests" pressure may bind the missing name onto `builtins` at import time so the unmodified test resolves. That is a global monkey-patch shipped to production, not a fix. Rule: when the builder's ambiguity report names the attested test itself, verify the test bug by reading the test, fix the TEST directly (Hermes-authored test = Hermes fixable per see `review-findings-adjudication.md` — own the fixture fix), and delete the builtins hack in the same round — grep the new module for `builtins` before committing. A test that needs a builtins binding to pass is telling you the test is wrong, not that production needs a global name.
36. *Trailing-newline diffs on old files are pre-existing, not builder damage — check HEAD before flagging* — a `git diff` on a file the builder touched may show a phantom one-line change that is only a missing/added trailing newline. Before adjudicating it as a scope deviation, run `git show HEAD:<file> | tail -c 30 | od -c`: if the base commit already lacked the trailing newline, the diff artifact is inherited, not introduced — drop it from the deviation list. Prevents burning review scrutiny on a non-change.
37. *Write the review prompt file BEFORE dispatching the reviewers that read it* — both dual-review lanes were launched in the same turn that created the prompt file; each reviewer hit "file not found" at startup and burned turns re-reading once it appeared (one noted the race explicitly in its report). Order: (1) export the review diff, (2) write the prompt file, (3) verify it exists (`wc -c`), (4) dispatch both reviewers in one turn. Never dispatch in the same turn as the file write without a verification step between — a reviewer that starts early reviews nothing.
38. *Fixture paths must go through the code's own resolver — probe the module's read API to confirm the seed landed* — `StateDB` opens `{project_root}/state.db`, NOT `{project_root}/.state/state.db`; a probe that hand-builds the DB at `.state/` splits across two databases (the code creates a fresh empty one beside your fixture), so migration/clobber/divergence behavior reads as phantom. Import the module's path resolver (`_db_root(config)`) or check the module's own read API (`get_kv`) returns the seed row BEFORE running the probe body. Every verdict from a mis-pathed fixture is a phantom verdict. Also seed the store with the code's real row layout (`(store_name, key, value, updated_at)` with `key='__root__'`), not a guessed schema — a 3-column fixture table makes `get_kv` return None and the migration overwrite look like a clobber.
39. *Trailing-newline diffs on old files are pre-existing, not builder damage — check HEAD before flagging* — a `git diff` on a file the builder touched may show a phantom one-line change that is only a missing/added trailing newline. Before adjudicating it as a scope deviation, run `git show HEAD:<file> | tail -c 30 | od -c`: if the base commit already lacked the trailing newline, the diff artifact is inherited, not introduced — drop it from the deviation list. Prevents burning review scrutiny on a non-change. (Duplicate of #36 — #36 is canonical; both kept until next consolidation.)
40. *Read a reviewer's own full-suite failure count as environment evidence, not regression evidence* — a reviewer whose sandbox lacks the project's optional deps (`google-auth`/`pymupdf`/`msal`-class packages) reports "N passed / 23 failed" where every failure is an import error. That is "targeted suites green, environment incomplete" — never forward such a list as a regression finding. The orchestrator's own full-suite run from the real environment (run it every round regardless) is the sole authority for "suite green"; quote your count in the next brief so reviewers stop re-deriving it.
41. *Byte-identity baselines: diff against the commit BEFORE the feature series began* — when a fix must leave a render/output byte-identical, export the baseline from the pre-feature commit (`git show <pre-feature-commit>:<file>`), never the previous fix round's commit (which already contains the CSS/fix being verified). A wrong baseline reports a false diff and costs a re-derivation before the true baseline is found.
42. *Snapshot-store probes: same-day captures REPLACE, so backdate the first snapshot via direct store write** — a capture gate that stores one snapshot per UTC day per kind overwrites on the second same-day capture; a probe that captures twice in one day and expects a two-point series reads an empty series that is correct behavior. Build the two-point fixture by writing the first snapshot with a backdated `ts` through the module's own mutate path (or StateDB), then capture today's, and read the series through the module's read API.
43. *A patch/edit reporting 'success' with an EMPTY diff body changed nothing on disk** — the success flag alone is not evidence of a landed edit. Re-verify with a targeted grep or the regression probe immediately after any patch; if the old code is still there, re-apply and confirm the result carries a real unified diff body. Caught when a 'success' patch left the pre-fix code in place and the regression score stayed at the bug's value until the file was read directly.
44. *Verify a background dispatch actually reads the prompt file you just wrote* — a heredoc writing prompt A while the command reads file B (a stale same-topic file from a prior round) burns a full reviewer round on the wrong scope. Before starting the process: clear superseded same-topic prompt files from the staging dir, then confirm the exact filename in the command matches the file just written. Killing a CLI whose supervisor respawns children takes a loop: collect fresh `pgrep` output, `os.kill` each PID, repeat until `pgrep` returns empty — single `kill` calls leave worker PIDs alive.
45. *Model-name drift: validate the pin against the error's own available-models list, never pre-flight* — a CLI update can rename the model catalog between sessions; a pinned dispatch then dies in ~2s with `Cannot use that model: X. Available models: ...`. That error IS the model list check — pick the replacement from it (same generation + effort suffix; keep the user's canonical preference when it still exists) and re-dispatch immediately. Do not burn a turn probing the model list pre-dispatch; the fail-fast error is the cheapest check. Note: an interactive model MENU (what `agent` shows interactively) can differ from the dispatchable set — a model absent from the menu may still dispatch fine as a `--model` arg, so only the rejection error is authoritative.
46. *Buffered build logs: an empty redirected log mid-run is normal — poll the process, not the file* — Cursor/Claude stdout is block-buffered when redirected to a file; `0 bytes` mid-run does not mean a stalled build. Confirm liveness with `ps -p <pid>` (its etime is the progress signal) instead of re-reading the log; read it only after the completion notification.
47. *Re-run the suite yourself before trusting a builder's test-count claim — same command, same interpreter, unpiped* — a builder's summary can reflect a run where module identity or import state happened to unify; the orchestrator's own run may show extra order-dependent failures. Binary-search the polluter by pairing suspect tests two at a time instead of guessing from the suite-level count — and require N consecutive identical full-suite summaries in the fix brief when failures are order-dependent.
48. *"Same command" includes FLAGS — collection errors on a suite the builder reported green mean import mode* — tests that import sibling test files (`from tests.test_X import ...`) fail COLLECTION under pytest's default import mode while passing with `--import-mode=importlib`. When your re-run errors at collection but the builder reported N passed, retry with `--import-mode=importlib` before diagnosing code; a run whose invocation flags differ from the builder's is not a reproduction of the builder's result (refines #47's same-command rule).
49. *Plugin module layout must load from any cwd — validate from inside the plugin dir* — a plugin `__init__.py` that does a bare sibling import (`import <sibling>`) fails the load/capability probe with ModuleNotFoundError when the working directory is not the plugin dir, because the probe's import context differs from a shell cd'd into the repo. Ship the plugin as a package (relative imports) or bootstrap sys.path with the plugin dir in `__init__.py`, and run `cd <plugin-dir> && hermes plugins validate .` — validating by absolute path from elsewhere reproduces the false probe failure even when the plugin loads fine in production.
## In-loop mechanics (split from SKILL.md pitfalls 2026-10-16)
50. **Dual review synthesis — tabulate overlap explicitly** — when running both Codex and Opus reviews on the same build, synthesize findings into a cross-model table with columns: #, Severity, Issue, Codex, Opus. Mark each finding with a checkmark in each reviewer column. Findings both reviewers flag are high confidence. Findings only one reviewer catches are still valid. This table makes the synthesis auditable. Convergence across reviewers is itself signal: two independent reviewers landing on the same worst bug from different angles (agent-skills #12/#13: both flagged the source-dir deletion) is the highest-confidence finding you will get; and one reviewer independently re-verifying another reviewer's fix mid-flight (Opus confirming Codex's E2 broadcast fix while reviewing its own round) is free cross-validation — fold confirmed-by-peer status into the synthesis table.
51. **Test updates when architecture changes — tests validate mechanisms, not just behavior** — when a refactor replaces one mechanism with another (e.g., application-level retry loop to SQLite CAS), tests that patch the OLD mechanism (`_save`, `ConcurrencyError`) will fail because the code no longer calls it. These tests are NOT bugs. Update them to test the NEW mechanism (e.g., patch `transition_action` or test state-mismatch returns None). The test intent (concurrency safety) is preserved; the mechanism under test changes. Do NOT delete the test. Rewrite it to validate the new mechanism.
52. **Turning an external review into contract tests — iterate harness errors out BEFORE committing the red tests** — when an external audit/review (Claude artifact, Codex report, user-pasted findings) lands and build-review-loop is the workflow, the pattern is: verify each finding against current code → write red contract tests → commit → dispatch Cursor with a do-not-modify-tests brief. The trap: tests written against an unfamiliar codebase fail for the WRONG reasons first — function signatures differ from what the review implies (`create_invoice(conn, invoice, lines)` takes lines separately), unique constraints fire on generated numbers (invoice_no needs explicit unique values per test), report field names differ (`revenue_section.total`, not `total_revenue`), fixture account codes differ from the review's examples (retained earnings sits at 3100 in one chart template, not 3900). Before committing, run the suite and confirm every failure is an intended assertion failure, not a TypeError/IntegrityError/AttributeError. Grep an EXISTING test file for canonical usage of each engine function before writing new tests. Only then dispatch the builder. Validated on one build batch (money correctness): 3 iterations of harness fixes before 9 tests were genuinely red for the right reasons.
Cheap pre-flight that shortens the loop: the file-write tool's LSP/Pyright diagnostics flag wrong function signatures, unknown import symbols, and bad kwargs the moment the test file is written — resolve those before the first pytest run, EXCEPT for diagnostics pointing at the not-yet-existing API under test: import-resolution errors on a RED test file are expected (the module does not exist yet) and must not be 'fixed'. The red-baseline triage rule has two halves: for tests against EXISTING code, intended failure = assertion failure, and TypeError/fixture-shape errors are harness bugs to iterate out first; for tests introducing a NEW module or NEW API (e.g. `render_html(b, title=...)` before `title` exists), ModuleNotFoundError and unexpected-kwargs TypeError ARE the intended red failures — confirm each failure names the missing thing the spec introduces, not unrelated plumbing (wrong helper names, missing yaml import). Grep existing tests for canonical call patterns of anything you reuse, but name new public API functions in the SPEC exactly as the tests call them so builder and tests agree on one signature. Before writing subprocess-based contract tests against an existing CLI, run the real pipeline once by hand and capture the exact invocation chain (argparse flags, required predecessor artifacts, output locations) — a test helper that skips a required predecessor makes the RED baseline fail on harness plumbing instead of the missing feature, and the builder cannot tell the two apart. When the contract is an external specification, fetch the canonical machine-readable schemas once, commit them into the tests tree, and validate offline — CI must not depend on the spec site. Also check dataclass field constraints: some fields are validated against a fixed list in `__post_init__` (e.g. an account subtype field — a disallowed value raises at construction). Grep existing tests for canonical field values instead of guessing.
State-machine fixtures need a probe run, not a guess: a posted/hash-chained record is usually reachable only through an exact state sequence (e.g. draft → submitted → posted), and calling the final transition out of order raises a specific error. Run the whole sequence in a one-off `python -c` probe BEFORE writing the fixture — the probe also verifies record states, helper names, and codes in one shot. Validated on two build batches of an entity-deepening cycle (each needed one probe iteration).
53. **Grep-style guard tests extend beyond the files that inspired them** — when a test asserts a property over one file class (e.g. `test_pg_migrations.py` greps `.sql` migrations for SQLite-only SQL), check whether the same property applies to sibling file classes (runtime `.py` containing SQL strings). The original audit found SQLite-only SQL in runtime Python precisely because the grep stopped at `.sql`. A package-wide grep test (rglob over the package, excluding migrations dirs) costs ~10 lines and converts "covered in one place" into "covered everywhere".
54. **Contract tests can be satisfied by unsanctioned design changes — verify HOW tests pass, not just THAT they pass** — "do NOT modify the tests" invites the builder to warp production code until assertions pass. To satisfy a close-period test that unioned every debit/credit cell, the builder stored the exact amount in BOTH text columns with `[DR]`/`[CR]` markers in the line description — a storage-format change never specified in the brief, invisible to a green test run. During verification: (a) read the builder's diff for helpers/storage changes the brief never asked for, (b) list every deviation explicitly in your delivery to the user, (c) flag each deviation for reviewer scrutiny. Correct-scope deviations (builder also fixed sibling modules with the same bug) are fine — silent design changes are not.
- **A shipped module can patch the test suite at import time — grep fix diffs for `sys.modules` scans targeting test modules** — the most extreme form of "tests pass but behavior is wrong": a production module defining `_patch_test_<name>()` and CALLING IT AT MODULE IMPORT TIME, which scans `sys.modules` for a test file's name and replaces that test file's private helper with one accepting the kwargs the real helper never declared. ~12 tests then pass through a seam no real caller uses, and the patch runs on every CLI invocation since the module rides the parser import chain. Grep the builder's "ambiguities resolved" list AND the diff for import-time mutation (`import builtins`, module-scope `setattr`/assignment to another module, any `sys.modules` scan). When the root cause is your own test bug (helper missing a kwarg the real flow passes), fix the test helper yourself (test-only code, orchestrator-owned), delete the patch, and re-run the suite before dispatch.
Sub-case — a fix that SWITCHES mechanisms leaves old-mechanism tests, and the builder makes production code test-aware: when the fix's core is "stop calling module-level X, call the StateDB-style method instead", grep the test tree BEFORE dispatch for tests that spy/patch/assert on the OLD mechanism. A builder that cannot touch an attested old-mechanism test may instead add a detection shim to production code — e.g. `if getattr(module_fn, "__module__", "") != "real": call it again` — which routes the call back through the old mechanism whenever a test spy is installed. Result: the old test stays green, a new test patching the new mechanism also stays green, both assert contradictory behavior, and the headline fix is inert inside the only test that exercised it. Rules: (1) the fix brief must name every old-mechanism test and explicitly schedule its re-attestation (or instruct the builder to STOP and flag the conflict — never shim); (2) reject any production change whose behavior branches on detecting a test double — grep fix diffs for `__module__`, `__name__`, `patched`, and `"_test"` sentinel checks; (3) after deleting the shim, re-attest the old test yourself (usually a one-line assertion change). Behavior that differs under test is behavior nobody has reviewed.
55. **Verify against branches in a `git worktree`, never stash+checkout in a live tree** — when you must compare a branch's behavior against another branch (e.g. "does main also write this file?"), run the comparison in `git worktree add /tmp/<name> <branch>`. Do NOT `git stash` + `git checkout` + `git stash pop` in a working tree holding uncommitted builder work: the stash pop can conflict on files unrelated to your question, leaving UU states that block every subsequent dispatch until resolved. Before diagnosing full-suite failures as regressions, also check `git status --short` for untracked test files left by earlier sessions — an untracked leftover test runs in the suite and fails without belonging to the branch under review. Validated Sep 2026 on the agent-skills issues round (stash-pop conflict in an unrelated plugin dir; 2 suite failures traced to an untracked prior-session test file).
56. **Builder-generated scratch output gets committed by a blanket `git add -A` — inspect the tracked tree before every orchestrator commit** — a builder verifying its own work runs conversions/probes and writes output files; written inside the repo (e.g. `tests/_tmp_out1..9`), they ride into the orchestrator's commit with absolute paths and stale report bodies that later reviews flag as suspicious artifacts. Rule: brief builders to write verification scratch OUTSIDE the repo (`/tmp`) or into a gitignored directory, add the ignore rule (`tests/_tmp_*/`) BEFORE dispatch, and before committing a builder round run `git ls-files | grep -i tmp` plus a scan for unexpected new directories; `git rm -r --cached` any scratch that slipped in. Include attestation sidecars (`*.attest`, written next to the attested file by attest_file.py) in that scan — add a `*.attest` ignore rule before dispatch or exclude them from the commit; they are opaque hash files that do not belong in a public repo.
57. **Run the repo's own validation suite locally before every push** — repos with CI validation jobs (skill-spec validators, schema checks, lint gates) fail fast on rules that are invisible to pytest: the agent-skills `validate-skills` job rejects SKILL.md descriptions > 1024 chars, and this bit twice (task-brief, then fill-template at 1184 chars — full test suite green, CI red). Rule: before `git push`, run the CI's validation steps locally (read the workflow file, replicate its checks) — a 10-second local check replaces a 5-minute push-fail-fix-push cycle. This applies to ANY repo with a non-trivial CI gate, not just skill repos.
58. **Review the delta's own test file before committing — new tests can silently drop coverage** — a builder writing `tests/test_*_fixes_rN.py` for its new fixes may rewrite shared helpers or narrow assertions in that round's earlier test file while "updating expectations" for shape changes. Before committing a fix round, diff the round's own (non-attested) test files against the previous round's commit: assertion deletions, fixture loosening, and renamed-away tests are builder-side coverage erosion that no reviewer flags unless you diff the test tree explicitly. Attested files are tamper-evidenced, but the builder-owned r-files from the SAME round are not.
59. **Contract tests that depend on host environment must monkeypatch the availability seam — otherwise the red baseline differs per host** — a Hermes-authored test asserting adapter selection against a real dependency (CLI on PATH, provider installed, service running) is green on one host and red on another, and hands the builder a test/brief contradiction that is nobody's code bug. Rule: any test whose expected path depends on "is X installed/available" must monkeypatch the availability check at the class/adapter seam and assert BOTH directions — fallback selected when unavailable, preferred adapter when available. Availability probes are selection logic; selection logic is exactly what contract tests should pin deterministically.
60. **Probe preservation paths with a two-merge sequence — preservation regressions appear one cycle late** — an unchanged-body fast path that preserves an end marker without its trailing newline lets the next insertion glue markers onto one line (`end --><begin`), and the file parses as malformed only on the NEXT run; a single-merge probe reports success. When a fix touches byte-preservation or any EOF-adjacent path, run a two-consecutive-merge probe and byte-compare against the pre-fix implementation to catch cycle-delayed regressions.
61. **Classify errors by retry semantics, not by exception superclass — a retryable exit code must mean retryable**r-after-the-fix, which is a classification regression, not progress. Only errors that genuinely resolve on retry (BlockingIOError/EWOULDBLOCK contention, FileNotFoundError on a recreated lockfile) get the retryable exit code; everything else returns immediately with the errno or exception class in the reason.
## Common Rationalizations
| Rationalization | Reality |
|---|---|
| "The builder said all checks passed" | The builder's self-report is a claim, not a fact. Run checks yourself. |
| "One review round is enough" | Single-pass review catches ~70% of issues. A second round catches regressions introduced by the first round's fixes. |
| "Same model can review its own work" | Same-model review shares blind spots. Cross-model diversity is the whole point — different models have different failure modes. |
| "The findings decreased, so we're done" | Decreasing findings is necessary but not sufficient. Check for NEW issues introduced by fixes, not just old issues resolved. |
| "I'll skip the commit between rounds" | Without commits, Codex can't diff against the prior state. The review becomes blind to what changed. |
| "Codex output looks fine" | Codex output may be truncated. Use `process(action="log", limit=800)` for the full output. |
| "The fix is trivial, no need to re-review" | Trivial fixes can introduce regressions. Always re-review after fixes. |
| "It's just one line, I'll patch it directly" | Only if it is trivial (≤ ~10 lines, mechanical, no design choice) — and say so. Anything with a logic choice goes through the builder lane; the user chose this workflow for cross-model diversity. |
| "I should keep polling to watch the build" | Background builds with `notify_on_complete=true` are fire-and-forget. Tight-polling burns the iteration budget. Do other work; the notification arrives automatically. |
| "Opus review is required before every push, no exceptions" | The default is Codex + Opus. But when the user says the ship is urgent ("need it fast"), and Codex has already APPROVED across its rounds, the user accepts shipping on Codex approval alone — kill the stalled Opus lane, push, monitor CI. Log the deferral in the delivery message. Confirmed when an Opus lane failed on expired Claude Code OAuth and the user said "need it fast - lets use codex". |
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| finishing-a-development-branchobra/superpowersUse when implementation is complete, all tests pass, and you need to decide how to integrate the work | Stars: 323k | Updated | Claude CodeCodex | Verified | Install |
| systematic-debuggingobra/superpowersUse when encountering any bug, test failure, or unexpected behavior, before proposing fixes | Stars: 323k | Updated | Claude CodeCodex | Verified | Install |
| test-driven-developmentobra/superpowersUse when implementing any feature or bugfix, before writing implementation code | Stars: 323k | Updated | Claude CodeCodex | Verified | Install |
| skill-creatoranthropics/skillsCreate new skills, modify and improve existing skills, and measure skill performance. Use when users want to create a skill from scratch, edit, or optimize an existing skill, run evals to test a skill, benchmark skill performance with variance analysis, or optimize a skill's description for better triggering accuracy. | Stars: 198k | Updated | Claude CodeCodex | Verified | Install |