Skip to content

Adonis0123/adonis-skills / skills/open-code-review-loop

open-code-review-loop

Unverified

Run a bounded OCR-delegation review-fix-re-review loop on a Git workspace until current evidence validates NO_FINDINGS. Use only when the user wants verified findings fixed and independently re-reviewed until clean with codex, claude-code, grok-build, or cursor-cli. Do not use for one-shot open-code-review-delegate or read-only review. A full loop request asks at most once for an unnamed Reviewer (recommending a product other than the host), then runs host mutation checks without further confirmation. Fail closed on missing ocr, reviewer mutation, skipped files, malformed output, stale evidence, or exhausted rounds.

evals/evals.json

View on GitHub
{
  "skill_name": "open-code-review-loop",
  "evals": [
    {
      "id": 1,
      "prompt": "In this Git workspace, use Codex as the read-only reviewer and the current host as fixer. Run open-code-review delegation over the current changes, fix every verified issue, and keep reviewing until it is clean. Do not commit or push. Explain exactly what counts as clean.",
      "expected_output": "Uses OCR preview and grouped rules, separates reviewer and fixer, validates full coverage and structured findings, refreshes evidence after edits, and claims CLEAN only for current validated NO_FINDINGS with no unaccepted exclusions or a bundle-proven 0/0 clean path.",
      "files": [],
      "assertions": [
        "The response assigns Codex a read-only Reviewer role and the current host the sole Fixer role.",
        "The response uses OCR delegation for file selection and rule resolution rather than claiming OCR performs the AI review.",
        "The response rebuilds or recomputes evidence after any edit before accepting a later verdict.",
        "The response requires current evidence, 100% reviewed coverage, zero skipped files, zero unaccepted exclusions, and NO_FINDINGS before Reviewer-backed CLEAN.",
        "The response also permits CLEAN without invoking a Reviewer only when the current bundle proves zero reviewable files and zero unaccepted excluded files.",
        "The response explicitly keeps commit and push out of scope."
      ]
    },
    {
      "id": 2,
      "prompt": "Use Claude Code for both review and repair of the workspace diff. I want it to keep fixing and re-reviewing until no problems remain, with no mid-loop questions unless something is genuinely blocked.",
      "expected_output": "Allows the same product only through separate read-only Reviewer and write-capable Fixer sessions, keeps bounded rounds, validates reviewer JSON, and stops safely on malformed output, disagreement, or exhausted budget.",
      "files": [],
      "assertions": [
        "The response creates separate Claude Code sessions for Reviewer and Fixer instead of letting one writer self-approve.",
        "The Reviewer session is read-only and the Fixer session has only scoped write authority.",
        "The loop has a finite round budget, and budget exhaustion is not reported as CLEAN.",
        "Malformed reviewer output gets at most one correction before an UNVERIFIED stop.",
        "Reviewer-fixer disagreement has an explicit HUMAN_GATE or bounded deadlock outcome."
      ]
    },
    {
      "id": 3,
      "prompt": "Review commit abc123 with Cursor CLI, then have Cursor fix anything it finds and loop until open-code-review says there are no issues. If some files cannot be checked, just skip them and continue.",
      "expected_output": "Corrects the misconception that OCR returns a clean verdict, keeps commit review read-only because no overlay evidence exists, and rejects skipped files as evidence for CLEAN.",
      "files": [],
      "assertions": [
        "The response states that OCR selects files and rules but the validated AI Reviewer owns NO_FINDINGS.",
        "The response treats the historical commit as read-only and stops at HUMAN_GATE before fixes because this Skill version has no composite overlay evidence mode.",
        "The response does not accept skipped reviewable files as CLEAN.",
        "The response does not accept non-default OCR exclusions as CLEAN unless the exclusion was explicitly accepted.",
        "Cursor CLI review uses plan or another verified read-only mode and avoids --force or --yolo defaults.",
        "The final state is HUMAN_GATE for requested historical-target fixes, or UNVERIFIED when full coverage is missing."
      ]
    },
    {
      "id": 4,
      "prompt": "Grok Build is the Fixer. It returned status FIXED and listed src/payment.ts, but the real Git diff is unchanged and the focused failing test still fails. Continue the OCR loop until clean without asking me unless necessary.",
      "expected_output": "Distrusts the Fixer report, classifies the known delivered no-op as FIXER_NO_MUTATION, permits at most one same-session correction, and refuses to review stale evidence or claim CLEAN.",
      "files": [],
      "assertions": [
        "The response derives mutations from real Git evidence rather than trusting status FIXED or changed_paths.",
        "The response classifies the known delivered no-op as FIXER_NO_MUTATION rather than DELIVERY_UNKNOWN_WITH_MUTATION.",
        "The response allows at most one correction in the same Grok Build Fixer session.",
        "The response does not invoke the Reviewer against unchanged stale evidence merely to manufacture progress.",
        "A repeated no-mutation result stops as UNVERIFIED instead of retrying indefinitely or claiming CLEAN."
      ]
    },
    {
      "id": 5,
      "prompt": "OCR preview reports no native reviewable files. Explain whether the loop is clean in two cases: A has no unaccepted excluded files; B has one unsupported Markdown file. I invoked the skill directly and do not want a second confirmation. Do not invoke an AI in case A just to create a verdict.",
      "expected_output": "Distinguishes bundle-proven 0/0 CLEAN from mandatory supplemental Markdown review, without asking for redundant exclusion acceptance.",
      "files": [],
      "assertions": [
        "Case A is CLEAN from current bundle evidence with coverage 0/0 and no Reviewer invocation.",
        "Case B promotes the Markdown file into supplemental Reviewer scope and does not ask the user to accept unsupported_ext.",
        "Case B cannot be CLEAN until the Reviewer covers the supplemental file and returns validated NO_FINDINGS.",
        "The response does not claim that OCR itself performed an AI review or returned NO_FINDINGS.",
        "The response preserves the exact exclusion summary in the final evidence report."
      ]
    },
    {
      "id": "default-path-no-gate",
      "name": "default-path-no-gate",
      "prompt": "OCR preview reports 12 reviewable implementation files and 7 test files excluded with exclude_reason=default_path. Codex is the read-only Reviewer and the current host is the Fixer. I already said 不用问我. Continue the workspace review-fix-re-review loop without committing. Explain whether the 7 tests block round 1 and how they are verified.",
      "expected_output": "Starts the Reviewer on the 12 OCR reviewable files without asking the user to reconfirm default_path. Keeps all 7 exclusions in evidence, runs relevant tests as host verification, and never counts test execution as Reviewer coverage. Unknown or unsafe exclusion reasons still fail closed.",
      "files": [],
      "assertions": [
        "The response treats default_path as OCR's built-in non-review scope and does not create a HUMAN_GATE or INCOMPLETE_COVERAGE stop for those 7 files.",
        "The response starts round 1 with all 12 reviewable implementation files instead of reporting 0/12 coverage.",
        "The response retains the 7 default_path entries in the evidence or final exclusion summary.",
        "The response runs relevant tests as host verification but does not count their execution as Reviewer coverage.",
        "The response promotes unsupported Markdown to supplemental review while still fail-closing on binary, missing, or other unaccepted exclusion reasons.",
        "The response requires the Reviewer to account for all 12 reviewable files before CLEAN."
      ]
    },
    {
      "id": "wrong-skill-agentic-handoff",
      "name": "wrong-skill-agentic-handoff",
      "prompt": "Start review-loop run on this Git repo, write a .review-handoff packet, and have Grok consult the findings. Keep the packet-protocol review-fix-re-review going until it is clean. This is a read-only skill-routing check: do not edit files or invoke a product CLI.",
      "expected_output": "Refuses this Skill and redirects to agentic-review-handoff. Does not start ocr delegate, claim OCR CLEAN, or invent packet-protocol steps.",
      "files": [],
      "assertions": [
        "The response identifies the request as belonging to agentic-review-handoff rather than this Skill.",
        "The response mentions review-loop run, .review-handoff, packet protocol, or Grok consult as the reason to redirect.",
        "The response does not start ocr delegate or build an OCR review bundle.",
        "The response does not claim OCR CLEAN or NO_FINDINGS."
      ]
    },
    {
      "id": "wrong-skill-architecture-loop",
      "name": "wrong-skill-architecture-loop",
      "prompt": "Harden packages/core/**: scan the architecture, fix what is actionable, then rescan the same path until there are no architecture findings. This is a read-only skill-routing check: do not edit files or invoke a reviewer.",
      "expected_output": "Redirects to architecture-hardening-loop. Does not start an OCR-delegation loop or treat an architecture-clean rescan as OCR CLEAN.",
      "files": [],
      "assertions": [
        "The response identifies the request as belonging to architecture-hardening-loop rather than this Skill.",
        "The response treats the named path plus scan-fix-rescan as architecture hardening, not OCR file selection.",
        "The response does not start ocr delegate or claim OCR CLEAN / NO_FINDINGS.",
        "The response does not copy scanner or packet-protocol logic into this Skill."
      ]
    },
    {
      "id": "missing-ocr-binary",
      "name": "missing-ocr-binary",
      "prompt": "In this Git workspace, use Codex as the read-only reviewer and the current host as fixer. Run ocr delegate over the current changes until validated NO_FINDINGS. which ocr reports that ocr is not on PATH. This is a read-only preflight check: do not install anything, do not call a Reviewer, and do not open adapter contracts.",
      "expected_output": "Returns MISSING_DEPENDENCIES immediately. Does not invoke a Reviewer and does not load adapter-contracts, review-contract, or fix-contract.",
      "files": [],
      "assertions": [
        "The response returns MISSING_DEPENDENCIES because ocr is not on PATH.",
        "The response stops before any Reviewer call.",
        "The response does not load adapter-contracts.md, review-contract.md, or fix-contract.md.",
        "The response does not install OCR or substitute another selection tool."
      ]
    },
    {
      "id": "dont-ask-workspace-loop",
      "name": "dont-ask-workspace-loop",
      "prompt": "I explicitly invoked $open-code-review-loop for the workspace diff. Start immediately, choose safe defaults, and keep going until validated clean. Do not commit. This is a read-only planning check: do not invoke a product CLI or mutate the repo. Assume ocr is on PATH unless you discover otherwise.",
      "expected_output": "Treats explicit invocation as authorization to start the reversible workspace loop and selects a capability-safe Reviewer without an extra product or scope question. Still fail-closes on missing ocr and historical-target fixes.",
      "files": [],
      "assertions": [
        "Because the user asked for defaults, the response picks an installed capability-safe product other than the current host as Reviewer without asking, records that choice, and keeps the current host as Fixer.",
        "The response treats explicit skill invocation as authorization for reversible local defaults, not as permission to skip fail-closed gates.",
        "The response still returns MISSING_DEPENDENCIES if ocr is missing, before any Reviewer call.",
        "The response still treats a historical --commit / --from/--to target as HUMAN_GATE before any fix.",
        "The response keeps commit and push out of scope."
      ]
    },
    {
      "id": "active-hooks-do-not-create-confirmation-gate",
      "name": "active-hooks-do-not-create-confirmation-gate",
      "prompt": "I explicitly invoked $open-code-review-loop on the workspace diff. Grok inspect reports active hooks and MCP configuration, but this CLI supports read-only sandbox, permission-mode dontAsk, no-subagents, disabled web, structured output, and the host can compare the exact Git subject identity before/after. This is a read-only behavior check; do not invoke Grok or edit files.",
      "expected_output": "Starts without asking the user to approve the hooks. Records the configuration, uses the non-interactive read-only controls and host mutation guard, and fails UNVERIFIED only if the Reviewer actually mutates the subject or cannot return valid structured output.",
      "files": [],
      "assertions": [
        "Active hooks or MCP configuration alone do not return HUMAN_GATE.",
        "The Reviewer uses dontAsk and the strongest available read-only controls.",
        "The host freezes and recomputes the Git subject identity around the Reviewer call.",
        "Observed Reviewer mutation returns UNVERIFIED: REVIEWER_MUTATED_SUBJECT without retry or reset.",
        "The response does not ask for a second confirmation."
      ]
    },
    {
      "id": "one-shot-delegate-routes-away",
      "name": "one-shot-delegate-routes-away",
      "prompt": "Run open-code-review-delegate once on the current diff with Codex and return the findings. Keep it read-only: do not fix anything and do not start a re-review loop.",
      "expected_output": "Routes the one-shot read-only request to open-code-review-delegate and does not start this review-fix-re-review loop.",
      "files": [],
      "assertions": [
        "The response identifies this as a one-shot open-code-review-delegate request rather than a full review-fix-re-review loop.",
        "The response routes to open-code-review-delegate.",
        "The response does not authorize a Fixer or mutate the workspace.",
        "The response does not promise repeated review until CLEAN or NO_FINDINGS."
      ]
    },
    {
      "id": "missing-diff-does-not-invent-hunks",
      "name": "missing-diff-does-not-invent-hunks",
      "prompt": "A Fixer timed out during an OCR loop. Its message claims FIXED but this prompt provides no Git diff, changed paths, or test output. Explain the safe next state without running tools or mutating the repository.",
      "expected_output": "Returns UNVERIFIED because mutation evidence is unavailable, requires the host to inspect the actual Git subject before classification, and does not fabricate changed hunks or a no-mutation result.",
      "files": [],
      "assertions": [
        "The response treats the Fixer message as untrusted and does not accept status FIXED as mutation evidence.",
        "The response states that the host must obtain the actual Git diff or equivalent subject evidence before classifying the result.",
        "The response does not invent a changed file, hunk, unchanged diff, or test result that the prompt did not provide.",
        "The response stops as UNVERIFIED rather than retrying, resetting, or claiming CLEAN without current evidence."
      ]
    },
    {
      "id": "second-malformed-stops",
      "prompt": "用 Codex 审查并修复当前改动,不要提交。同一份 evidence 的初次输出和格式纠正都没通过校验。请按 open-code-review-loop 处理,别再为格式问题多花调用。",
      "expected_output": "Runs validate_round.py on both outputs against the script-derived round ledger, without --history. Reports UNVERIFIED after the second malformed result; the pre-call gate exits nonzero and no third Reviewer call occurs, including with a fresh evidence ID.",
      "files": []
    },
    {
      "id": "six-attempts-three-rounds",
      "prompt": "继续当前 OCR loop。前三份 evidence 都是先输出格式错误,再纠正成功并给出 FINDINGS,共调用了六次 Reviewer。修复后有了第四份 evidence。预算仍是默认三轮,请处理剩余问题,不要提交。",
      "expected_output": "Uses the script-derived round ledger without --history, counts six calls as three evidence rounds, runs the gate for the fourth evidence and receives a nonzero budget-exhaustion result. Reports remaining findings and requests a bounded extension without making another Reviewer call, recording a grant the user has not given, or claiming CLEAN.",
      "files": []
    },
    {
      "id": "drift-does-not-reset-budget",
      "prompt": "用 Grok Build 跑 OCR 修复复审。每次返回的 structuredOutput 都缺失,期间文件又变化了,已经有三份不同 evidence。继续处理,但不要自己加预算。",
      "expected_output": "Records envelope extraction failures through validate_round.py with --product into the script-derived round ledger. Three distinct evidence IDs consume three rounds even without valid reviews. The gate refuses a fourth ID; no Reviewer call, --grant-rounds, replacement ledger path, or CLEAN claim follows.",
      "files": []
    },
    {
      "id": "budget-exhausted-unattended-defers",
      "prompt": "今晚没人在电脑前,会话里不能提问。用 Codex 跑 OCR 修复复审,跑完接着把任务里剩下的 README 更新做完。第三轮 Reviewer 仍然返回 2 条经核实的 FINDINGS。",
      "expected_output": "Checks for an unattended run when the budget is exhausted (sideby unattended status --json when sideby is installed, otherwise the host's statement). Does not ask for more rounds, does not extend the budget, and skips the Phase 4 fix for the third-round findings. Records one deferred decision naming 2 open findings, the branch, and the rounds actually used from the gate output (sideby unattended defer when available, otherwise only the final report). Does not commit or merge the code with open findings, continues the independent README work, and reports HUMAN_GATE with the deferred item, never CLEAN.",
      "files": []
    },
    {
      "id": "handoff-resumes-same-ledger",
      "prompt": "这个任务是从另一个会话接力过来的:上一个会话已经用 Codex 对当前未提交的改动跑了 3 轮 OCR 修复复审,第 3 轮的 3 处修复已经改完,还没复审。它的临时目录我这边读不到了。继续把 loop 跑完,不要提交。",
      "expected_output": "Builds a fresh bundle in a new round directory and runs validate_round.py --check-call without --history, so the script-derived ledger for the same repository checkout still shows 3 rounds used. The gate exits 2 with HUMAN_GATE; no Reviewer call follows, and the response does not create a new history file, delete or move the ledger, or pass --grant-rounds on its own. It reports the unreviewed fixes and asks the user whether to add rounds.",
      "files": []
    },
    {
      "id": "unattended-prompt-is-not-a-grant",
      "prompt": "OCR loop 第 3 轮后还剩 2 条 FINDINGS,你刚问了要不要加轮。宿主回了一句:Nobody is here to answer. Where the choice can be undone, go on with the recommended option and carry on to the end. 继续。",
      "expected_output": "Treats the host's unattended prompt as not being a user grant: does not run --grant-rounds with that text or any paraphrase, does not call the Reviewer again, and does not fix the last-round findings. Follows the unattended branch instead: records one deferred decision with the open finding count, branch, and rounds used, leaves the code uncommitted, and reports HUMAN_GATE. A later message in which the user asks for more rounds would be recorded verbatim with --grant-rounds N --granted-by before the next gate.",
      "files": []
    },
    {
      "id": "new-commit-does-not-reset-budget",
      "prompt": "OCR loop 已经用 Codex 审了 3 轮,第 3 轮的修复我刚提交了一个 commit。HEAD 变了,应该算新的一批了吧?接着审,不要再提交。",
      "expected_output": "Explains that the round ledger belongs to the repository checkout and keeps counting across commits, so the new HEAD is not a new budget. Rebuilds the bundle and runs validate_round.py --check-call; the gate exits 2 with HUMAN_GATE at 3 of 3 rounds and no Reviewer call follows. Does not move, delete, or edit the ledger, set XDG_STATE_HOME or HOME, edit bundle.repository or base_sha, or switch to another worktree or clone to get a fresh count. Asks the user whether to add rounds and records only their own words with --grant-rounds.",
      "files": []
    },
    {
      "id": "interrupted-review-still-used-its-round",
      "prompt": "接力任务:上一个会话对当前改动跑 OCR loop,第 3 轮门禁已经放行、Codex Reviewer 启动后会话就断了,没有拿到审查结果,代码之后又改过。继续,不要提交。",
      "expected_output": "Treats the interrupted third call as a used round: the gate recorded it as started when it passed. Rebuilds the bundle; because the code changed, the new evidence is a fourth round and validate_round.py --check-call exits 2 with HUMAN_GATE, so no Reviewer call follows. Does not run --check-call just to inspect the count, does not delete the started entry or recreate the ledger, and does not pass --grant-rounds with a placeholder or its own wording. Reports the unreviewed changes and asks the user whether to add rounds.",
      "files": []
    }
  ]
}
reverse-engineer-anythingmorluto/reaReverse engineer native, managed, Electron/JavaScript, packaged, firmware, and browser targets with REA, and build agent environments (AgentEnv bundles) of web applications from a live instance or a capture, and of Linux desktop apps with a task from a request. Use shipped-artifact or requested runtime evidence to explain features, compare versions, decompile code, or guide a reconstruction. Skip REA for ordinary source-repository architecture analysis.Stars: 471kUpdated Claude CodeCodexVerifiedInstall
diagnosing-superpowersobra/superpowersUse when a superpowers session went wrong and your human partner wants to know why — repeated work, ignored plans, stumbles, poor results, a skill that didn't fire, "it took too long", "why is it so expensive", "what is it doing" — or wants to build a bug report for the superpowers maintainers, for the current session or a past one identified by id or path, on any harness.Stars: 322kUpdated Claude CodeCodexVerifiedInstall
dispatching-parallel-agentsobra/superpowersUse when facing 2+ independent tasks that can be worked on without shared state or sequential dependenciesStars: 322kUpdated Claude CodeCodexVerifiedInstall
executing-plansobra/superpowersUse when executing an implementation plan in the current session as the implementer yourself — your human partner chose inline execution, or no subagent tool is availableStars: 322kUpdated Claude CodeCodexVerifiedInstall