Skip to content

Lambenthan/empiricalwiki / i18n/en/skills/review

review

Verified

General-purpose cross-model review — Review LLM independently reviews any research artifact, outputs structured scores, wiki entity mapping, and improvement suggestions

/review

Review any research artifact (idea, proposal, experiment plan, paper draft, claim) using cross-model review. Uses Review LLM as an independent reviewer. Outputs a structured score, actionable improvement suggestions, and a mapping to wiki entities (which claims need strengthening, which gaps are discovered). Supports three difficulty levels (standard / hard / adversarial) and four review focuses. Can be used standalone or called by /ideate, /refine, /exp-design.

Inputs

  • artifact: the artifact to review, one of:
    • slug of a wiki page (e.g. sparse-lora-for-edge-devices, searched in ideas/experiments/claims/)
    • file path (e.g. wiki/outputs/paper-draft-v1.md)
    • free text (directly pasted proposal or idea description)
  • --difficulty (optional, default standard):
    • standard: single-round review, delivers structured feedback
    • hard: multi-round dialogue (up to 3 rounds), Claude rebuts each weakness
    • adversarial: multi-round dialogue (up to 3 rounds), Review LLM additionally attempts to find fatal flaws, simulating the harshest reviewer
  • --focus (optional, default comprehensive review):
    • method: focus on technical correctness, novelty, and feasibility of method design
    • evidence: focus on sufficiency of evidence, experimental rigor, claim support
    • writing: focus on clarity, structural organization, and argumentative logic
    • completeness: focus on missing content (related work, ablations, baselines)

Outputs

  • Review Report (output to terminal):
    • Overall Score (1-10)
    • Strengths (list of positives)
    • Weaknesses (list of issues, ranked by severity)
    • Questions (reviewer questions)
    • Actionable Suggestions (improvement suggestions ranked by priority)
    • Wiki Entity Mapping (which claims need strengthening, which gaps were found)
    • Verdict: ready / needs-work / major-revision / rethink
  • If --difficulty >= hard: additionally includes multi-round dialogue history and final revised score
  • This skill does not directly modify the wiki, but outputs a list of suggested wiki updates

Wiki Interaction

Reads

  • wiki/papers/*.md — locate papers cited by the artifact, verify citation accuracy
  • wiki/concepts/*.md — understand technical concepts involved in the artifact
  • wiki/claims/*.md — check the current status and confidence of claims the artifact depends on
  • wiki/experiments/*.md — find related experiment results
  • wiki/ideas/*.md — if reviewing an idea, check its context
  • wiki/graph/context_brief.md — global context
  • wiki/graph/open_questions.md — check completeness against the gap map
  • .claude/skills/shared-references/cross-model-review.md — reviewer independence principle

Writes

  • None. Review is a read-only query operation.
    • Review results are output to terminal; the user or caller (e.g. /refine) decides whether to apply them.

Graph edges created

  • None.

Workflow

Precondition: confirm working directory is the wiki project root (containing wiki/, raw/, tools/).

Step 1: Load Context

  1. Parse artifact:

    • If slug: search sequentially in wiki/ideas/, wiki/experiments/, wiki/claims/, wiki/papers/, wiki/outputs/ for {slug}.md
    • If file path: read directly
    • If free text: use directly
  2. Determine artifact type: idea / experiment / claim / paper-draft / proposal / other

  3. Load relevant wiki context:

    • Read wiki/graph/context_brief.md for global perspective
    • Read wiki/graph/open_questions.md for knowledge gap list
    • Load relevant wiki pages by artifact type:
      • idea → its origin_gaps claims, related papers
      • experiment → its target_claim, related experiments
      • claim → its evidence sources, related papers and experiments
      • paper-draft → all wiki pages it cites
  4. Read cross-model-review.md: confirm Review LLM independence principle

  5. Build reviewer system prompt (based on --focus):

    Base prompt (all focuses):

    You are a senior ML researcher reviewing a research artifact.
    Be thorough, specific, and constructive. For every weakness, suggest a concrete fix.
    Score on a 1-10 scale where:
    - 1-3: Fundamental flaws, not salvageable in current form
    - 4-5: Significant issues but core idea may have merit
    - 6-7: Solid work with clear areas for improvement
    - 8-9: Strong work, minor issues only
    - 10: Exceptional, publication-ready
    

    Focus-specific additions:

    • method: additionally assess technical correctness, novelty of approach, feasibility, comparison to alternatives
    • evidence: additionally assess experimental rigor, statistical significance, claim-evidence alignment, missing controls
    • writing: additionally assess clarity, logical flow, notation consistency, figure quality, related work coverage
    • completeness: additionally assess missing baselines, missing ablations, missing datasets, missing related work, reproducibility

    Adversarial addition (adversarial mode only):

    Additionally: actively search for fatal flaws. A fatal flaw is anything that,
    if true, would make the entire contribution invalid (incorrect proof, data leakage,
    unfair comparison, published prior work). If you find one, flag it clearly.
    

Step 2: Review LLM Initial Review

Follow cross-model-review.md: do not send any of Claude's pre-judgments to Review LLM.

mcp__llm-review__chat:
  system: {reviewer system prompt from Step 1}
  message: |
    ## Artifact to Review
    {artifact full text}

    ## Context from Knowledge Base
    {relevant wiki context: related claims with status/confidence, related experiments, gap map entries}

    ## Review Instructions
    Please provide:
    1. **Strengths** (3-5 bullet points)
    2. **Weaknesses** (ranked by severity, each with a concrete suggestion to fix)
    3. **Questions** (things that are unclear or need clarification)
    4. **Score** (1-10 with one-sentence justification)
    5. **Verdict**: ready / needs-work / major-revision / rethink
    6. **Claim-level feedback**: For each claim referenced in the artifact, assess whether the evidence is sufficient. List any claims that need stronger support.
    7. **Knowledge gaps identified**: Any open questions or missing knowledge that would strengthen this work.

Record the threadId returned by Review LLM (for multi-round dialogue in Step 3).

Step 3: Multi-Round Dialogue (hard / adversarial mode)

Skip this step if --difficulty is standard.

Respond to each of Review LLM's weaknesses (up to 3 rounds):

Round N (N = 1, 2, 3):

  1. Claude analyzes Review LLM's weaknesses and classifies each:

    • Rebuttal: Claude has strong reasoning or wiki evidence to counter it → write a rebuttal
    • Acknowledge: the weakness genuinely exists → acknowledge it and propose a fix
    • Clarify: the weakness is based on a misunderstanding → provide clarification
  2. Send Claude's response to Review LLM:

    mcp__llm-review__chat-reply:
      threadId: {from Step 2}
      message: |
        Thank you for the review. Here are my responses:
    
        {for each weakness: rebuttal / acknowledgment / clarification}
    
        Please re-evaluate considering these responses. Update your score if warranted.
        If --difficulty == adversarial: Also, please try harder to find any remaining
        fatal flaws I may have missed.
    
  3. Review LLM responds with a new assessment and revised score

  4. If Review LLM's score change < 0.5 and no new weaknesses → stop dialogue (converged)

  5. If 3 rounds reached → stop dialogue

Step 4: Structured Output

Synthesize Step 2 + Step 3 results into a structured Review Report:

# Review Report: {artifact title}

## Meta
- **Artifact type**: {idea / experiment / claim / paper-draft / proposal}
- **Difficulty**: {standard / hard / adversarial}
- **Focus**: {method / evidence / writing / completeness / comprehensive}
- **Reviewer**: Review LLM (configured in `.env`)
- **Rounds**: {1 for standard, N for hard/adversarial}

## Score: {final score}/10 — {verdict}

| Verdict | Meaning |
|---------|---------|
| ready | Ready to use or submit directly |
| needs-work | Clear improvement points; usable after fixes |
| major-revision | Core sections need significant revision |
| rethink | Fundamental direction may be flawed; reconsider |

## Strengths
1. {strength 1}
2. {strength 2}
...

## Weaknesses (by severity)

### Critical
- {weakness}: {specific description} → **Fix**: {specific fix suggestion}

### Major
- {weakness}: {specific description} → **Fix**: {specific fix suggestion}

### Minor
- {weakness}: {specific description} → **Fix**: {specific fix suggestion}

## Questions
1. {question}
...

## Wiki Entity Mapping

### Claims needing stronger support
| Claim | Current confidence | Issue | Suggested action |
|-------|-------------------|-------|------------------|
| [[claim-slug]] | 0.6 | Evidence is indirect | Run targeted experiment |

### Knowledge gaps identified
| Gap | Related to | Suggested action |
|-----|-----------|------------------|
| {description} | [[slug]] | /ingest, /exp-run, or /query |

### Suggested wiki updates
- `wiki/claims/{slug}.md`: update confidence, add evidence note
- `wiki/ideas/{slug}.md`: add risk factor from review
- `wiki/graph/open_questions.md`: will be updated on next rebuild

## Dialogue History (hard/adversarial only)

### Round 1
**Review LLM**: {summary of initial review}
**Claude**: {summary of rebuttals/acknowledgments}

### Round 2
**Review LLM**: {updated assessment}
...

## Actionable Items (ranked)
1. [CRITICAL] {action item}
2. [MAJOR] {action item}
3. [MINOR] {action item}

Constraints

  • Reviewer independence: strictly follow shared-references/cross-model-review.md; do not leak Claude's pre-judgments to Review LLM
  • Do not modify wiki: review only outputs suggestions; it does not directly modify any wiki pages. Wiki modifications are handled by the caller (e.g. /refine)
  • Scores must have justification: scores without a rationale are not accepted
  • Weaknesses must have fixes: every weakness must include a specific, actionable fix suggestion; vague criticism is not accepted
  • Claim-level mapping is required: output must include the Wiki Entity Mapping section, mapping review findings to specific wiki entities
  • Adversarial mode must search for fatal flaws: e.g. fully published identical work, incorrect proofs, data leakage
  • Multi-round dialogue capped at 3 rounds: prevents infinite loops; output current state if 3 rounds do not converge
  • Use [[slug]] when referencing wiki pages: all references to wiki pages use wikilink syntax

Error Handling

  • Artifact not found: prompt user to check slug or path, list likely candidate pages
  • Review LLM unavailable: downgrade to Claude self-review mode; annotate report with "single-model review, cross-model verification unavailable"; recommend the user retry with Review LLM later
  • Wiki empty: proceed with review normally, but annotate Wiki Entity Mapping section with "wiki empty, no entity mapping available"
  • Artifact too long: if it exceeds Review LLM's context window, review section by section and merge at the end
  • Review LLM returns invalid response: retry once; if still invalid, use Claude self-review fallback
  • Review LLM does not converge in multi-round dialogue: force-stop after 3 rounds; output the last round's score and summary

Dependencies

Tools(via Bash)

  • No direct tool calls (review does not require deterministic tools)

MCP Servers

  • mcp__llm-review__chat — Review LLM initial review (Step 2)
  • mcp__llm-review__chat-reply — Review LLM multi-round dialogue (Step 3)

Claude Code Native

  • Read — read artifact and wiki pages
  • Glob — find wiki page corresponding to artifact

Shared References

  • .claude/skills/shared-references/cross-model-review.md — reviewer independence principle (required reading)

Called by

  • /ideate Phase 4 (review top ideas)
  • /refine each iteration round (review current version)
  • /exp-design --review (review experiment plan)
add-product-e2e-evalpaperclipai/paperclipAdd or extend a Paperclip full-stack runner E2E workflow, fixture, matcher, or report evidence path for local or Daytona execution.Stars: 115kUpdated Claude CodeCodexVerifiedInstall
add-runner-evalpaperclipai/paperclipAdd or extend a Paperclip Runner protocol evaluation definition, roster, assertion, or report fixture with provenance and narrow validation.Stars: 115kUpdated Claude CodeCodexVerifiedInstall
diagnose-why-work-stoppedpaperclipai/paperclipCompatibility entry point for failed, stalled, looping, or over-recovered Paperclip work. Use fix-failed-paperclip-run for the full diagnosis, regression, repair, real verification, and green-PR workflow; honor an explicitly requested diagnosis-only scope.Stars: 115kUpdated Claude CodeCodexVerifiedInstall
fix-failed-paperclip-runpaperclipai/paperclipDiagnose and fix failed, stuck, slow, or incorrectly blocked Paperclip tasks and runs. Use with a task/run URL or failure report to trace the root cause, prove a regression red then green, check second-order effects, verify the real workflow, and deliver a PR with passing checks.Stars: 115kUpdated Claude CodeCodexVerifiedInstall