Lambenthan/empiricalwiki / i18n/en/skills/review
review
VerifiedGeneral-purpose cross-model review — Review LLM independently reviews any research artifact, outputs structured scores, wiki entity mapping, and improvement suggestions
/review
Review any research artifact (idea, proposal, experiment plan, paper draft, claim) using cross-model review. Uses Review LLM as an independent reviewer. Outputs a structured score, actionable improvement suggestions, and a mapping to wiki entities (which claims need strengthening, which gaps are discovered). Supports three difficulty levels (standard / hard / adversarial) and four review focuses. Can be used standalone or called by /ideate, /refine, /exp-design.
Inputs
artifact: the artifact to review, one of:- slug of a wiki page (e.g.
sparse-lora-for-edge-devices, searched in ideas/experiments/claims/) - file path (e.g.
wiki/outputs/paper-draft-v1.md) - free text (directly pasted proposal or idea description)
- slug of a wiki page (e.g.
--difficulty(optional, defaultstandard):standard: single-round review, delivers structured feedbackhard: multi-round dialogue (up to 3 rounds), Claude rebuts each weaknessadversarial: multi-round dialogue (up to 3 rounds), Review LLM additionally attempts to find fatal flaws, simulating the harshest reviewer
--focus(optional, default comprehensive review):method: focus on technical correctness, novelty, and feasibility of method designevidence: focus on sufficiency of evidence, experimental rigor, claim supportwriting: focus on clarity, structural organization, and argumentative logiccompleteness: focus on missing content (related work, ablations, baselines)
Outputs
- Review Report (output to terminal):
- Overall Score (1-10)
- Strengths (list of positives)
- Weaknesses (list of issues, ranked by severity)
- Questions (reviewer questions)
- Actionable Suggestions (improvement suggestions ranked by priority)
- Wiki Entity Mapping (which claims need strengthening, which gaps were found)
- Verdict:
ready/needs-work/major-revision/rethink
- If
--difficulty >= hard: additionally includes multi-round dialogue history and final revised score - This skill does not directly modify the wiki, but outputs a list of suggested wiki updates
Wiki Interaction
Reads
wiki/papers/*.md— locate papers cited by the artifact, verify citation accuracywiki/concepts/*.md— understand technical concepts involved in the artifactwiki/claims/*.md— check the current status and confidence of claims the artifact depends onwiki/experiments/*.md— find related experiment resultswiki/ideas/*.md— if reviewing an idea, check its contextwiki/graph/context_brief.md— global contextwiki/graph/open_questions.md— check completeness against the gap map.claude/skills/shared-references/cross-model-review.md— reviewer independence principle
Writes
- None. Review is a read-only query operation.
- Review results are output to terminal; the user or caller (e.g. /refine) decides whether to apply them.
Graph edges created
- None.
Workflow
Precondition: confirm working directory is the wiki project root (containing wiki/, raw/, tools/).
Step 1: Load Context
-
Parse artifact:
- If slug: search sequentially in
wiki/ideas/,wiki/experiments/,wiki/claims/,wiki/papers/,wiki/outputs/for{slug}.md - If file path: read directly
- If free text: use directly
- If slug: search sequentially in
-
Determine artifact type: idea / experiment / claim / paper-draft / proposal / other
-
Load relevant wiki context:
- Read
wiki/graph/context_brief.mdfor global perspective - Read
wiki/graph/open_questions.mdfor knowledge gap list - Load relevant wiki pages by artifact type:
- idea → its origin_gaps claims, related papers
- experiment → its target_claim, related experiments
- claim → its evidence sources, related papers and experiments
- paper-draft → all wiki pages it cites
- Read
-
Read cross-model-review.md: confirm Review LLM independence principle
-
Build reviewer system prompt (based on --focus):
Base prompt (all focuses):
You are a senior ML researcher reviewing a research artifact. Be thorough, specific, and constructive. For every weakness, suggest a concrete fix. Score on a 1-10 scale where: - 1-3: Fundamental flaws, not salvageable in current form - 4-5: Significant issues but core idea may have merit - 6-7: Solid work with clear areas for improvement - 8-9: Strong work, minor issues only - 10: Exceptional, publication-readyFocus-specific additions:
method: additionally assess technical correctness, novelty of approach, feasibility, comparison to alternativesevidence: additionally assess experimental rigor, statistical significance, claim-evidence alignment, missing controlswriting: additionally assess clarity, logical flow, notation consistency, figure quality, related work coveragecompleteness: additionally assess missing baselines, missing ablations, missing datasets, missing related work, reproducibility
Adversarial addition (adversarial mode only):
Additionally: actively search for fatal flaws. A fatal flaw is anything that, if true, would make the entire contribution invalid (incorrect proof, data leakage, unfair comparison, published prior work). If you find one, flag it clearly.
Step 2: Review LLM Initial Review
Follow cross-model-review.md: do not send any of Claude's pre-judgments to Review LLM.
mcp__llm-review__chat:
system: {reviewer system prompt from Step 1}
message: |
## Artifact to Review
{artifact full text}
## Context from Knowledge Base
{relevant wiki context: related claims with status/confidence, related experiments, gap map entries}
## Review Instructions
Please provide:
1. **Strengths** (3-5 bullet points)
2. **Weaknesses** (ranked by severity, each with a concrete suggestion to fix)
3. **Questions** (things that are unclear or need clarification)
4. **Score** (1-10 with one-sentence justification)
5. **Verdict**: ready / needs-work / major-revision / rethink
6. **Claim-level feedback**: For each claim referenced in the artifact, assess whether the evidence is sufficient. List any claims that need stronger support.
7. **Knowledge gaps identified**: Any open questions or missing knowledge that would strengthen this work.
Record the threadId returned by Review LLM (for multi-round dialogue in Step 3).
Step 3: Multi-Round Dialogue (hard / adversarial mode)
Skip this step if --difficulty is standard.
Respond to each of Review LLM's weaknesses (up to 3 rounds):
Round N (N = 1, 2, 3):
-
Claude analyzes Review LLM's weaknesses and classifies each:
- Rebuttal: Claude has strong reasoning or wiki evidence to counter it → write a rebuttal
- Acknowledge: the weakness genuinely exists → acknowledge it and propose a fix
- Clarify: the weakness is based on a misunderstanding → provide clarification
-
Send Claude's response to Review LLM:
mcp__llm-review__chat-reply: threadId: {from Step 2} message: | Thank you for the review. Here are my responses: {for each weakness: rebuttal / acknowledgment / clarification} Please re-evaluate considering these responses. Update your score if warranted. If --difficulty == adversarial: Also, please try harder to find any remaining fatal flaws I may have missed. -
Review LLM responds with a new assessment and revised score
-
If Review LLM's score change < 0.5 and no new weaknesses → stop dialogue (converged)
-
If 3 rounds reached → stop dialogue
Step 4: Structured Output
Synthesize Step 2 + Step 3 results into a structured Review Report:
# Review Report: {artifact title}
## Meta
- **Artifact type**: {idea / experiment / claim / paper-draft / proposal}
- **Difficulty**: {standard / hard / adversarial}
- **Focus**: {method / evidence / writing / completeness / comprehensive}
- **Reviewer**: Review LLM (configured in `.env`)
- **Rounds**: {1 for standard, N for hard/adversarial}
## Score: {final score}/10 — {verdict}
| Verdict | Meaning |
|---------|---------|
| ready | Ready to use or submit directly |
| needs-work | Clear improvement points; usable after fixes |
| major-revision | Core sections need significant revision |
| rethink | Fundamental direction may be flawed; reconsider |
## Strengths
1. {strength 1}
2. {strength 2}
...
## Weaknesses (by severity)
### Critical
- {weakness}: {specific description} → **Fix**: {specific fix suggestion}
### Major
- {weakness}: {specific description} → **Fix**: {specific fix suggestion}
### Minor
- {weakness}: {specific description} → **Fix**: {specific fix suggestion}
## Questions
1. {question}
...
## Wiki Entity Mapping
### Claims needing stronger support
| Claim | Current confidence | Issue | Suggested action |
|-------|-------------------|-------|------------------|
| [[claim-slug]] | 0.6 | Evidence is indirect | Run targeted experiment |
### Knowledge gaps identified
| Gap | Related to | Suggested action |
|-----|-----------|------------------|
| {description} | [[slug]] | /ingest, /exp-run, or /query |
### Suggested wiki updates
- `wiki/claims/{slug}.md`: update confidence, add evidence note
- `wiki/ideas/{slug}.md`: add risk factor from review
- `wiki/graph/open_questions.md`: will be updated on next rebuild
## Dialogue History (hard/adversarial only)
### Round 1
**Review LLM**: {summary of initial review}
**Claude**: {summary of rebuttals/acknowledgments}
### Round 2
**Review LLM**: {updated assessment}
...
## Actionable Items (ranked)
1. [CRITICAL] {action item}
2. [MAJOR] {action item}
3. [MINOR] {action item}
Constraints
- Reviewer independence: strictly follow
shared-references/cross-model-review.md; do not leak Claude's pre-judgments to Review LLM - Do not modify wiki: review only outputs suggestions; it does not directly modify any wiki pages. Wiki modifications are handled by the caller (e.g. /refine)
- Scores must have justification: scores without a rationale are not accepted
- Weaknesses must have fixes: every weakness must include a specific, actionable fix suggestion; vague criticism is not accepted
- Claim-level mapping is required: output must include the Wiki Entity Mapping section, mapping review findings to specific wiki entities
- Adversarial mode must search for fatal flaws: e.g. fully published identical work, incorrect proofs, data leakage
- Multi-round dialogue capped at 3 rounds: prevents infinite loops; output current state if 3 rounds do not converge
- Use [[slug]] when referencing wiki pages: all references to wiki pages use wikilink syntax
Error Handling
- Artifact not found: prompt user to check slug or path, list likely candidate pages
- Review LLM unavailable: downgrade to Claude self-review mode; annotate report with "single-model review, cross-model verification unavailable"; recommend the user retry with Review LLM later
- Wiki empty: proceed with review normally, but annotate Wiki Entity Mapping section with "wiki empty, no entity mapping available"
- Artifact too long: if it exceeds Review LLM's context window, review section by section and merge at the end
- Review LLM returns invalid response: retry once; if still invalid, use Claude self-review fallback
- Review LLM does not converge in multi-round dialogue: force-stop after 3 rounds; output the last round's score and summary
Dependencies
Tools(via Bash)
- No direct tool calls (review does not require deterministic tools)
MCP Servers
mcp__llm-review__chat— Review LLM initial review (Step 2)mcp__llm-review__chat-reply— Review LLM multi-round dialogue (Step 3)
Claude Code Native
Read— read artifact and wiki pagesGlob— find wiki page corresponding to artifact
Shared References
.claude/skills/shared-references/cross-model-review.md— reviewer independence principle (required reading)
Called by
/ideatePhase 4 (review top ideas)/refineeach iteration round (review current version)/exp-design --review(review experiment plan)
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| add-product-e2e-evalpaperclipai/paperclipAdd or extend a Paperclip full-stack runner E2E workflow, fixture, matcher, or report evidence path for local or Daytona execution. | Stars: 115k | Updated | Claude CodeCodex | Verified | Install |
| add-runner-evalpaperclipai/paperclipAdd or extend a Paperclip Runner protocol evaluation definition, roster, assertion, or report fixture with provenance and narrow validation. | Stars: 115k | Updated | Claude CodeCodex | Verified | Install |
| diagnose-why-work-stoppedpaperclipai/paperclipCompatibility entry point for failed, stalled, looping, or over-recovered Paperclip work. Use fix-failed-paperclip-run for the full diagnosis, regression, repair, real verification, and green-PR workflow; honor an explicitly requested diagnosis-only scope. | Stars: 115k | Updated | Claude CodeCodex | Verified | Install |
| fix-failed-paperclip-runpaperclipai/paperclipDiagnose and fix failed, stuck, slow, or incorrectly blocked Paperclip tasks and runs. Use with a task/run URL or failure report to trace the root cause, prove a regression red then green, check second-order effects, verify the real workflow, and deliver a PR with passing checks. | Stars: 115k | Updated | Claude CodeCodex | Verified | Install |