Skip to content

Lambenthan/empiricalwiki / .claude/skills/review

review

Verified

通用跨模型审查:Review LLM 对任意研究制品进行独立评审,输出结构化评分、wiki 实体映射与改进建议

FileKindSize
SKILL.mdskill md11.9 KB
---
description: 通用跨模型审查:Review LLM 对任意研究制品进行独立评审,输出结构化评分、wiki 实体映射与改进建议
argument-hint: <artifact-path-or-slug> [--difficulty standard|hard|adversarial] [--focus method|evidence|writing|completeness]
---

# /review

> 对任意研究制品(idea、proposal、experiment plan、paper draft、claim)进行跨模型审查。
> 使用 Review LLM 作为独立审稿人,输出结构化评分、可操作的改进建议,以及与 wiki 实体的映射
> (哪些 claims 需要加强,哪些 gaps 被发现)。
> 支持三种难度级别(standard / hard / adversarial)和四种审查焦点。
> 可独立使用,也被 /ideate、/refine、/exp-design 调用。

## Inputs

- `artifact`:要审查的制品,以下之一:
  - wiki 页面的 slug(如 `sparse-lora-for-edge-devices`,从 ideas/experiments/claims/ 中查找)
  - 文件路径(如 `wiki/outputs/paper-draft-v1.md`)
  - 自由文本(直接粘贴的 proposal 或 idea 描述)
- `--difficulty`(可选,默认 `standard`):
  - `standard`:单轮审查,给出结构化反馈
  - `hard`:多轮对话(最多 3 轮),Claude 对每个 weakness 进行 rebuttal
  - `adversarial`:多轮对话(最多 3 轮),Review LLM 额外尝试找致命缺陷,模拟最严苛的审稿人
- `--focus`(可选,默认全面审查):
  - `method`:聚焦方法设计的正确性、创新性、可行性
  - `evidence`:聚焦证据是否充分、实验是否严谨、claim 是否 well-supported
  - `writing`:聚焦表达清晰度、结构组织、论证逻辑
  - `completeness`:聚焦是否遗漏关键内容(相关工作、ablation、baseline)

## Outputs

- **Review Report**(输出到终端):
  - Overall Score(1-10)
  - Strengths(优点列表)
  - Weaknesses(缺点列表,按严重程度排序)
  - Questions(审稿人的疑问)
  - Actionable Suggestions(可操作的改进建议,按优先级排序)
  - Wiki Entity Mapping(哪些 claims 需要加强,哪些 gaps 被发现)
  - Verdict:`ready` / `needs-work` / `major-revision` / `rethink`
- 若 `--difficulty >= hard`:额外包含多轮对话记录和最终修正后的评分
- 该 skill **不直接修改 wiki**,但会输出建议的 wiki 更新列表

## Wiki Interaction

### Reads
- `wiki/papers/*.md` — 查找制品引用的论文,验证引用正确性
- `wiki/concepts/*.md` — 理解制品涉及的技术概念
- `wiki/claims/*.md` — 检查制品依赖的 claims 当前状态和 confidence
- `wiki/experiments/*.md` — 查找相关实验结果
- `wiki/ideas/*.md` — 如果审查的是 idea,检查其上下文
- `wiki/graph/context_brief.md` — 获取全局上下文
- `wiki/graph/open_questions.md` — 对照 gap map 检查完整性
- `.claude/skills/shared-references/cross-model-review.md` — 审稿独立性原则

### Writes
- **无**。Review 是只读查询操作。
  - 审查结果输出到终端,由用户或调用方(如 /refine)决定是否应用。

### Graph edges created
- **无**。

## Workflow

**前置**:确认工作目录为 wiki 项目根(包含 `wiki/`、`raw/`、`tools/` 的目录)。

### Step 1: 加载上下文

1. **解析 artifact**:
   - 若为 slug:按顺序在 `wiki/ideas/`、`wiki/experiments/`、`wiki/claims/`、`wiki/papers/`、`wiki/outputs/` 中查找 `{slug}.md`
   - 若为文件路径:直接读取
   - 若为自由文本:直接使用
2. **确定 artifact 类型**:idea / experiment / claim / paper-draft / proposal / other
3. **加载相关 wiki 上下文**:
   - 读取 `wiki/graph/context_brief.md` 获取全局视角
   - 读取 `wiki/graph/open_questions.md` 获取知识缺口列表
   - 根据 artifact 类型,加载相关 wiki 页面:
     - idea → 其 origin_gaps 对应的 claims,相关 papers
     - experiment → 其 target_claim,相关 experiments
     - claim → 其 evidence 来源,相关 papers 和 experiments
     - paper-draft → 其引用的所有 wiki 页面
4. **读取 cross-model-review.md**:确认 Review LLM 独立性原则
5. **构建 reviewer system prompt**(根据 --focus):

   **Base prompt(所有 focus):**
   ```
   You are a senior ML researcher reviewing a research artifact.
   Be thorough, specific, and constructive. For every weakness, suggest a concrete fix.
   Score on a 1-10 scale where:
   - 1-3: Fundamental flaws, not salvageable in current form
   - 4-5: Significant issues but core idea may have merit
   - 6-7: Solid work with clear areas for improvement
   - 8-9: Strong work, minor issues only
   - 10: Exceptional, publication-ready
   ```

   **Focus-specific additions:**
   - `method`:额外要求评估 technical correctness, novelty of approach, feasibility, comparison to alternatives
   - `evidence`:额外要求评估 experimental rigor, statistical significance, claim-evidence alignment, missing controls
   - `writing`:额外要求评估 clarity, logical flow, notation consistency, figure quality, related work coverage
   - `completeness`:额外要求评估 missing baselines, missing ablations, missing datasets, missing related work, reproducibility

   **Adversarial addition(仅 adversarial 模式):**
   ```
   Additionally: actively search for fatal flaws. A fatal flaw is anything that,
   if true, would make the entire contribution invalid (incorrect proof, data leakage,
   unfair comparison, published prior work). If you find one, flag it clearly.
   ```

### Step 2: Review LLM 首轮审查

**遵循 cross-model-review.md**:不向 Review LLM 发送任何 Claude 的预判。

```
mcp__llm-review__chat:
  system: {reviewer system prompt from Step 1}
  message: |
    ## Artifact to Review
    {artifact full text}

    ## Context from Knowledge Base
    {relevant wiki context: related claims with status/confidence, related experiments, gap map entries}

    ## Review Instructions
    Please provide:
    1. **Strengths** (3-5 bullet points)
    2. **Weaknesses** (ranked by severity, each with a concrete suggestion to fix)
    3. **Questions** (things that are unclear or need clarification)
    4. **Score** (1-10 with one-sentence justification)
    5. **Verdict**: ready / needs-work / major-revision / rethink
    6. **Claim-level feedback**: For each claim referenced in the artifact, assess whether the evidence is sufficient. List any claims that need stronger support.
    7. **Knowledge gaps identified**: Any open questions or missing knowledge that would strengthen this work.
```

记录 Review LLM 返回的 `threadId`(用于 Step 3 多轮对话)。

### Step 3: 多轮对话(hard / adversarial 模式)

若 `--difficulty` 为 `standard`,跳过此步。

**对 Review LLM 的每个 weakness 进行回应**(最多 3 轮):

**Round N(N = 1, 2, 3):**

1. Claude 分析 Review LLM 的 weaknesses,对每个 weakness 分类:
   - **可反驳(rebuttal)**:Claude 有充分理由或 wiki 证据反驳 → 写出 rebuttal
   - **承认(acknowledge)**:weakness 确实存在 → 承认并提出修复方案
   - **需要更多信息(clarify)**:weakness 基于误解 → 提供澄清

2. 将 Claude 的回应发送给 Review LLM:
   ```
   mcp__llm-review__chat-reply:
     threadId: {from Step 2}
     message: |
       Thank you for the review. Here are my responses:

       {for each weakness: rebuttal / acknowledgment / clarification}

       Please re-evaluate considering these responses. Update your score if warranted.
       If --difficulty == adversarial: Also, please try harder to find any remaining
       fatal flaws I may have missed.
   ```

3. Review LLM 回应新的评估和修正后的评分

4. 若 Review LLM 的评分变化 < 0.5 且无新 weakness → 停止对话(收敛)
5. 若已达 3 轮 → 停止对话

### Step 4: 结构化输出

综合 Step 2 + Step 3 结果,生成结构化 Review Report:

```markdown
# Review Report: {artifact title}

## Meta
- **Artifact type**: {idea / experiment / claim / paper-draft / proposal}
- **Difficulty**: {standard / hard / adversarial}
- **Focus**: {method / evidence / writing / completeness / 全面}
- **Reviewer**: Review LLM
- **Rounds**: {1 for standard, N for hard/adversarial}

## Score: {final score}/10 — {verdict}

| Verdict | 含义 |
|---------|------|
| ready | 可直接使用或提交 |
| needs-work | 有明确的改进点,修复后可用 |
| major-revision | 核心部分需要重大修改 |
| rethink | 基本方向可能有问题,需重新考虑 |

## Strengths
1. {strength 1}
2. {strength 2}
...

## Weaknesses (by severity)

### Critical
- {weakness}: {具体描述} → **Fix**: {具体修复建议}

### Major
- {weakness}: {具体描述} → **Fix**: {具体修复建议}

### Minor
- {weakness}: {具体描述} → **Fix**: {具体修复建议}

## Questions
1. {question}
...

## Wiki Entity Mapping

### Claims needing stronger support
| Claim | Current confidence | Issue | Suggested action |
|-------|-------------------|-------|------------------|
| [[claim-slug]] | 0.6 | Evidence is indirect | Run targeted experiment |

### Knowledge gaps identified
| Gap | Related to | Suggested action |
|-----|-----------|------------------|
| {描述} | [[slug]] | /ingest, /exp-run, or /query |

### Suggested wiki updates
- `wiki/claims/{slug}.md`: update confidence, add evidence note
- `wiki/ideas/{slug}.md`: add risk factor from review
- `wiki/graph/open_questions.md`: will be updated on next rebuild

## Dialogue History (hard/adversarial only)

### Round 1
**Review LLM**: {summary of initial review}
**Claude**: {summary of rebuttals/acknowledgments}

### Round 2
**Review LLM**: {updated assessment}
...

## Actionable Items (ranked)
1. [CRITICAL] {action item}
2. [MAJOR] {action item}
3. [MINOR] {action item}
```

## Constraints

- **审稿独立性**:严格遵循 `shared-references/cross-model-review.md`,不向 Review LLM 泄露 Claude 的预判
- **不修改 wiki**:review 只输出建议,不直接修改任何 wiki 页面。wiki 修改由调用方(如 /refine)执行
- **score 必须有 justification**:不接受没有理由的分数
- **weakness 必须有 fix**:每个 weakness 必须附带具体的、可操作的修复建议,不接受空洞批评
- **claim-level mapping 必须**:输出必须包含 Wiki Entity Mapping 部分,将 review 发现映射到具体 wiki 实体
- **adversarial 模式必须搜索致命缺陷**:如已发表的完全相同工作、证明错误、数据泄露等
- **多轮对话最多 3 轮**:防止无限循环,若 3 轮后仍未收敛则以当前状态输出
- **引用 wiki 页面时使用 [[slug]]**:所有对 wiki 页面的引用使用 wikilink 语法

## Error Handling

- **artifact 找不到**:提示用户检查 slug 或路径,列出可能的候选页面
- **Review LLM 不可用**:降级为 Claude 自我审查模式,报告标注「single-model review, cross-model verification unavailable」,建议用户稍后用 Review LLM 重新审查
- **wiki 为空**:正常执行审查,但 Wiki Entity Mapping 部分标注「wiki empty, no entity mapping available」
- **artifact 太长**:若超过 Review LLM 上下文窗口,按 section 分段审查,最后合并
- **Review LLM 返回无效响应**:重试一次,若仍无效则使用 Claude 自审降级方案
- **多轮对话中 Review LLM 不收敛**:3 轮后强制结束,输出最后一轮的评分和总结

## Dependencies

### Tools(via Bash)
- 无直接工具调用(review 不需要确定性工具)

### MCP Servers
- `mcp__llm-review__chat` — Review LLM 首轮审查(Step 2)
- `mcp__llm-review__chat-reply` — Review LLM 多轮对话(Step 3)

### Claude Code Native
- `Read` — 读取 artifact 和 wiki 页面
- `Glob` — 查找 artifact 对应的 wiki 页面

### Shared References
- `.claude/skills/shared-references/cross-model-review.md` — 审稿独立性原则(必读)

### Called by
- `/ideate` Phase 4(审查 top ideas)
- `/refine` 每轮迭代(审查当前版本)
- `/exp-design --review`(审查实验计划)
receiving-code-reviewobra/superpowersUse when receiving code review feedback, before implementing suggestions, especially if feedback seems unclear or technically questionable - requires technical rigor and verification, not performative agreement or blind implementationStars: 322kUpdated Claude CodeCodexVerifiedInstall
requesting-code-reviewobra/superpowersUse when completing tasks, implementing major features, or before merging to verify work meets requirementsStars: 322kUpdated Claude CodeCodexVerifiedInstall
using-git-worktreesobra/superpowersUse when starting feature work that needs isolation from current workspace or before executing implementation plans - ensures an isolated workspace exists via native tools or git worktree fallbackStars: 322kUpdated Claude CodeCodexVerifiedInstall
verification-before-completionobra/superpowersUse when about to claim work is complete, fixed, or passing, before committing or creating PRs - requires running verification commands and confirming output before making any success claims; evidence before assertions alwaysStars: 322kUpdated Claude CodeCodexVerifiedInstall