Skip to content

moonlight-lupin/agent-skills / plugins/dev-pipeline/skills/test-evals

test-evals

Verified

Use when writing tests or verifying behavior before code lands — the stage that writes the plan's tests and runs them RED before build; TDD red-green-refactor, the iron law (no production code without a failing test first), seam-first public-interface tests, test sizes, mock discipline. Or when the user says "TDD N loops each batch Opus loop", "batch properly, Opus on each batch before commit", "implement, use TDD", or wants multi-feature TDD with per-batch Opus review before commit. Supports both Hermes-builds and Cursor-builds. Covers pre-commit five-axis review, severity labels, and verifying external review findings before fixing.

FileKindSize
SKILL.mdexecutableskill md14.1 KB
references/batched-tdd-opus-loop.mdexecutablescript19.0 KB
references/key-rules.mdexecutablescript3.4 KB
references/pitfalls-cli-and-env-mechanics.mdexecutablescript6.3 KB
references/pitfalls-loop-convergence.mdexecutablescript11.6 KB
references/pitfalls-review-findings.mdexecutablescript13.0 KB
references/pre-commit-review.mdexecutablescript17.0 KB
references/requesting-code-review.mdexecutablescript9.2 KB
references/review-report-shape.mdexecutablescript6.6 KB
references/test-driven-development.mdexecutablescript12.2 KB

references/pitfalls-cli-and-env-mechanics.md

View on GitHub
# CLI and environment mechanics

Read before dispatching a build or review, opening a PR, or diagnosing a CI miss. Timing, turn caps, gh failures, and pre-existing CI pins live here.

## Contents

- Pitfall rules
- gh PR creation failing
- Pre-existing CI failures in multiple locations
- `delegate_task` subagent review is a viable Opus alternative
- Opus 5.5 high takes 5–8 minutes
- Claude Code print-mode turn exhaustion
- Opus multi-file review: name exactly which files to read, demand immediate write, use --max-turns 25
- Parallel dispatch: Codex review + Cursor build for different files

## Pitfall rules

### gh PR creation failing

4. **gh PR creation failing** — fine-grained PATs may lack
   `repository.defaultBranchRef` scope. Don't retry endlessly — tell the user
   to create the PR from GitHub UI.

### Pre-existing CI failures in multiple locations

6. **Pre-existing CI failures in multiple locations** — a version-pin test
   can fail in both the test file (`tests/test_api.py`) AND the CI workflow
   YAML (`.github/workflows/ci.yml`) with hardcoded regex patterns. When
   fixing a pre-existing CI failure, `grep -rn "<old-version>" tests/
   .github/` to find ALL hardcoded references. A single fix in the test
   file won't make CI green if the workflow YAML has its own inline Python
   check with the same hardcoded pattern. See
   `references/ci-pre-existing-failures.md` for the fastmcp version-pin
   example.

### `delegate_task` subagent review is a viable Opus alternative

10. **`delegate_task` subagent review is a viable Opus alternative** — when
    the Cursor CLI is unavailable or the diff includes untracked new files
    that `git diff` misses, use `delegate_task` with an `output_schema` to
    dispatch a subagent review. The subagent can read the actual codebase on
    disk (not just the diff), cross-check the diff against the working tree,
    and find issues the diff alone misses. In the Tablina semantic-cache+HyDE
    session, the subagent noticed the on-disk code had already diverged from
    the diff (a pre-emptive ACL fix) and correctly flagged the diff version
    as BLOCKING while confirming the working tree was safe. Pass enough
    context in the `goal` field — architecture facts, security boundaries,
    the specific review focus areas — because the subagent starts with zero
    conversation context. Write findings to a file AND return them via
    `output_schema` so you get structured results even if the file write fails.

### Opus 5.5 high takes 5–8 minutes

13. **Opus 5.5 high takes 5–8 minutes** — the `claude-opus-5-5-high`
    model is thorough but slow. Don't panic at the wait or kill the process
    prematurely. Set `notify_on_complete=true` and continue other work (e.g.
    fixing findings from the Codex review that already completed) while
    waiting. If the user asks for a specific model and you dispatch the wrong
    one (e.g. `claude-opus-5-medium` instead of `claude-opus-5-5-high`),
    kill the process and relaunch with the correct model — the user will
    notice the model name in the output.

### Claude Code print-mode turn exhaustion

14. **Claude Code print-mode turn exhaustion** — Opus 5 in `claude -p` mode
    always tries to use tools (Read, Bash) on turn 1, even when the diff is
    piped via stdin and `--allowedTools ""` is set. With `--max-turns N`,
    it exhausts N turns reading files and exits with "Reached max turns"
    without producing any review text. Symptom: exit code 1, output file
    contains only "Error: Reached max turns (N)". This happens because
    Opus 5's default behavior is to explore the codebase before answering —
    it doesn't realize the content is already in its context via stdin.
    **Fix**: (a) set `--max-turns 1` so it can't loop, (b) set
    `--allowedTools ""` to block tool access, (c) include in the prompt:
    "Do NOT use any tools. The entire diff is provided below in this prompt.
    Just read it and write your review directly as text output.", (d) use
    `--effort medium` not `high` (high makes the exploration instinct
    stronger). With `--max-turns 1` and `--allowedTools ""`, the model has
    no choice but to answer from context. Cost: ~$0.63 for a 1671-line
    diff (5 commits). Do NOT use `--bare` mode — it breaks OAuth auth
    ("Not logged in · Please run /login"). Claude Code CAN run as root;
    OAuth auth works fine via `claude auth status`.

### Opus multi-file review: name exactly which files to read, demand immediate write, use --max-turns 25

20. **Opus multi-file review: name exactly which files to read, demand immediate write, use --max-turns 25** — when reviewing a multi-file diff with Claude Code (`claude -p --model opus`), Opus tries to explore the entire codebase and exhausts `--max-turns` without writing output. The working pattern for multi-file reviews:
    ```bash
    claude -p "$(cat /tmp/opus-review-prompt.md)" \
      --model opus --allowedTools "Read,Write,Bash" --max-turns 25
    ```
    The prompt MUST:
    - Name exactly which files to read (e.g. "Read ONLY these 4 files: state_store.py, webhook_security.py, ...")
    - Demand immediate write: "Write your review IMMEDIATELY to /tmp/opus-review-phaseN.md"
    - Specify the output format (Verdict, BLOCKING, MAJOR, MINOR, Summary)
    - Cap files to read (e.g. "Do NOT read more than 5 files")
    **Use `--max-turns 25`, not 10.** Opus reads 1 file per turn (each Read call = 1 turn). With 4 files to review, that's 4 turns for reading + 1 for git diff + 1 for pytest = 6 turns before it can write. `--max-turns 10` was too low for any multi-file review — Opus hit the limit every time without writing output. 25 is the minimum for multi-file reviews. This was proven across CoS Phases 1-4C (2026-08-14/15): with max-turns 10, Opus failed 3/5 reviews (exhausted turns, no output file written). With max-turns 25, Opus succeeded 100% and produced detailed reviews (10-19K chars each).

### Parallel dispatch: Codex review + Cursor build for different files

21. **Parallel dispatch: Codex review + Cursor build for different files** — when Loop 1 review (Codex) and Loop 2 build (Cursor) touch different source files, dispatch both simultaneously. Codex reviews the Loop 1 diff (already committed) while Cursor builds Loop 2 (new files). This saves 3-5 minutes per phase. Verify file non-overlap before dispatching — if both touch the same file, serialize instead.
finishing-a-development-branchobra/superpowersUse when implementation is complete, all tests pass, and you need to decide how to integrate the workStars: 322kUpdated Claude CodeCodexVerifiedInstall
systematic-debuggingobra/superpowersUse when encountering any bug, test failure, or unexpected behavior, before proposing fixesStars: 322kUpdated Claude CodeCodexVerifiedInstall
test-driven-developmentobra/superpowersUse when implementing any feature or bugfix, before writing implementation codeStars: 322kUpdated Claude CodeCodexVerifiedInstall
skill-creatoranthropics/skillsCreate new skills, modify and improve existing skills, and measure skill performance. Use when users want to create a skill from scratch, edit, or optimize an existing skill, run evals to test a skill, benchmark skill performance with variance analysis, or optimize a skill's description for better triggering accuracy.Stars: 198kUpdated Claude CodeCodexVerifiedInstall