Lambenthan/empiricalwiki / i18n/en/skills/rebuttal
rebuttal
VerifiedParse review comments → atomize concerns (Rvx-Cy) → map to wiki claims → check evidence → Review LLM stress-test → generate rebuttal
/rebuttal
Parse review comments, atomize each concern (Rvx-Cy numbering) and map it to a wiki claim, check whether evidence is sufficient (tracing back to wiki experiments), simulate reviewer follow-up questions with Review LLM (stress-test, scored 1-5), and generate a formal plain-text rebuttal and a rich-text rebuttal. Safety checks ensure no fabrication, no overpromise, full coverage.
Inputs
review: source of review comments, one of:- file path (e.g.
raw/reviews/reviewer1.txt,raw/reviews/meta-review.md) - multiple file paths (comma-separated:
raw/reviews/R1.txt,raw/reviews/R2.txt,raw/reviews/R3.txt) - directly pasted review text
- file path (e.g.
--paper-slug(optional): slug of the associated paper in wiki/outputs/, used to locate PAPER_PLAN--venue(optional): target conference/journal (ICLR / NeurIPS / ICML / ACL / CVPR); affects rebuttal format and word limits--stress-test(optional, enabled by default): Review LLM simulates reviewer follow-up; disable with--no-stress-test--format(optional, defaultformal): output formatformal: formal plain-text rebuttal (suitable for pasting directly into submission system)rich: rich-text version (with wiki [[links]], detailed analysis, improvement plan)
Outputs
- wiki/outputs/rebuttal-{slug}.md — rich-text rebuttal (with [[wikilinks]], evidence tracing, analysis tables)
- wiki/outputs/rebuttal-{slug}.txt — formal rebuttal (plain text, suitable for pasting into submission system)
- wiki/claims/*.md — if a concern exposes an evidence gap, append a suggestion to
## Open questions - wiki/log.md — append log entry
Wiki Interaction
Reads
wiki/claims/*.md— map concerns to claims, check evidence sufficiencywiki/experiments/*.md— find experiment results supporting claimswiki/papers/*.md— find citation context for referenced paperswiki/concepts/*.md— understand the conceptual background of method-related concernswiki/ideas/*.md— find the motivation and pilot results for ideaswiki/outputs/PAPER_PLAN.md— understand paper structure (from /paper-plan, if --paper-slug provided)wiki/graph/context_brief.md— global contextwiki/graph/edges.jsonl— claim-experiment-paper relationships.claude/skills/shared-references/cross-model-review.md— Review LLM stress-test independence
Writes
wiki/outputs/rebuttal-{slug}.md— rich-text versionwiki/outputs/rebuttal-{slug}.txt— formal plain-text versionwiki/claims/*.md— append reviewer-identified gaps to## Open questions(do not directly modify confidence/status; add suggestions only)wiki/log.md— append log entry
Graph edges created
- None (rebuttal is a query operation; does not modify the knowledge graph)
Workflow
Precondition:
- Confirm working directory is the wiki project root (containing
wiki/,raw/,tools/) - Read
cross-model-review.mdto confirm stress-test independence principle - Generate slug:
python3 tools/research_wiki.py slug "{paper-slug}-rebuttal"
Step 1: Parse Review Comments
-
Read review text:
- If file path(s): read all specified files
- If direct text: use directly
- Merge multiple reviewers' comments, annotated by source (Reviewer 1/2/3/Meta)
-
Identify structure:
- Extract each reviewer's: overall score (Accept/Reject/Borderline), confidence, summary, Strengths, Weaknesses, questions
- If the format is non-standard (plain text), use LLM to parse into structured format
-
Output: structured comments for each reviewer
Step 2: Atomize Concerns
Split each weakness and question into independent atomic concerns:
-
Splitting rules:
- A single weakness may be a compound sentence containing multiple independent concerns ("the method lacks ablation experiments and also does not compare with X" → split into 2 concerns)
- Assign each atomic concern an ID in
Rvx-Cyformat (Rv1-C1 = Reviewer 1, Concern 1; Rv1-C2 = Reviewer 1, Concern 2) - Retain reviewer number to ensure traceability back to the original comment
-
Classify each concern:
- evidence: factual questions about experimental data or result interpretation
- method: methodological questions about method design or algorithmic correctness
- missing: missing experiments/analysis/comparisons/citations
- clarity: unclear expression, symbol confusion, figure issues
- scope: insufficient contribution, applicability questions
- novelty: overlap with existing work, insufficient innovation
- minor: formatting, typos, and other small issues
-
Assess severity: critical / major / minor
-
Output: atomized concern list, each containing {id (Rvx-Cy), reviewer, type, severity, text}
Step 3: Map Concerns to Wiki Claims
For each concern:
-
Find associated claim:
- Extract keywords from concern text
- Search
wiki/claims/*.mdfor matching claims - Read
wiki/graph/edges.jsonlto find claim-experiment relationships - If no direct match is found: annotate as "unmapped" (no direct claim correspondence)
-
Check Evidence Status:
- Read the claim's evidence list
- Count strong/moderate/weak evidence
- Find results of associated experiments
- Judgment:
- Sufficient: strong >= 1 or moderate >= 2
- Partial: evidence exists but insufficient strength
- Insufficient: no evidence or only weak evidence
- Contradicted: evidence of invalidates type exists
-
Output:
| Concern ID | Reviewer | Type | Severity | Claim mapped | Evidence Status | Strategy |
|---|---|---|---|---|---|---|
| Rv1-C1 | R1 | method | critical | [[claim-slug]] | sufficient | A |
| Rv1-C2 | R1 | missing | major | [[claim-slug]] | insufficient | B |
| Rv2-C1 | R2 | novelty | major | unmapped | — | D |
Step 4: Draft Rebuttal Responses
Draft a response for each concern according to its strategy:
Strategy A — Evidence sufficient (respond directly):
- Cite specific experiment results and data (annotate source, ensure traceability to wiki/experiments/)
- Point to evidence in the wiki (convert to paper citations)
- If the concern is based on a misunderstanding: politely clarify, point to the relevant Section in the paper
Strategy B — Evidence insufficient (acknowledge + concrete plan):
- Honestly acknowledge that current evidence is not sufficient
- Propose a concrete supplementary experiment plan (can link to /exp-design)
- State a specific timeline and resource requirements
- Do not use vague commitments; only commit to concrete executable supplementary experiments
Strategy C — Clarity issue (commit to revision):
- Acknowledge the unclear expression
- Provide the improved description (show the revised text directly in the rebuttal)
- List specific Paper Edit plans
Strategy D — Scope/Novelty challenge (argue):
- Highlight essential differences from existing work
- Cite novelty-check results (if available)
- Point out differences the reviewer may have overlooked
Format for each response:
**[Rvx-Cy]** {concern summary}
{response text, 2-5 sentences, annotated sources for traceability}
Safety checks (per response):
- No fabrication: do not fabricate data or experiment results
- No overpromise: only commit to specific executable supplementary experiments
- Cited data is recorded in wiki/experiments/
- If claim is challenged/deprecated, do not pretend it is supported
Step 5: Review LLM Stress-Test
Follow cross-model-review.md: do not send Claude's rebuttal strategy analysis to Review LLM.
If --stress-test is enabled (default):
mcp__llm-review__chat:
system: "You are a critical reviewer who has just read a rebuttal to your review
comments. You are skeptical and will push back on weak responses.
For each rebuttal response, assess on a scale of 1-5:
1 = unconvincing (deflection or fabrication suspected)
2 = weak (vague, no concrete evidence)
3 = acceptable (addresses concern but could be stronger)
4 = strong (concrete evidence, clear reasoning)
5 = fully convincing (compelling evidence, thorough response)
Also check for overpromise: are commitments specific and feasible?
Provide a follow-up question for any response scoring <= 3."
message: |
## Original Review Concerns
{atomic concerns list with Rvx-Cy IDs}
## Author Rebuttal
{drafted rebuttal responses}
## Please assess each response (score 1-5) and provide follow-up questions.
Handle Review LLM feedback:
- Score 4-5 (convincing): keep original response
- Score 3 (acceptable): strengthen response, add details suggested by Review LLM
- Score 1-2 (unconvincing/weak): rewrite response, consider switching strategy (A→B, acknowledge insufficiency)
Second round (if any responses scored <= 2):
mcp__llm-review__chat-reply:
threadId: {previous thread}
message: |
We've revised the following responses:
{revised responses}
Please re-assess (score 1-5).
Maximum 2 rounds of stress-test. Handle follow-up questions and update responses.
Step 6: Format Output + Safety Check
6a. Format formal rebuttal-{slug}.txt (plain text, suitable for submission system):
We thank the reviewers for their constructive feedback. We address each concern below.
Reviewer 1:
[Rv1-C1] {concern summary}
{response}
[Rv1-C2] {concern summary}
{response}
Reviewer 2:
...
Summary of Revisions:
- {bulleted list of planned changes}
Additional Experiments (if applicable):
- {new experiments committed to, with timeline}
6b. Format rich-text rebuttal-{slug}.md:
# Rebuttal Analysis: {paper title}
## Coverage Summary
| Concern ID | Type | Severity | Claim | Evidence Status | Review LLM Score | Strategy |
|------------|------|----------|-------|-----------------|------------|----------|
| Rv1-C1 | method | critical | [[claim-slug]] | sufficient | 4/5 | A |
| Rv1-C2 | missing | major | [[claim-slug]] | insufficient | 3/5 | B |
## Responses
### Reviewer 1
**[Rv1-C1]** ...
**[Rv1-C2]** ...
## Evidence Gap Analysis
| Claim | Confidence | Gap | Needed |
|-------|-----------|-----|--------|
| [[claim-slug]] | 0.5 | No ablation on dataset X | Run ablation experiment |
## Action Items
### Paper Edits
| Section | Change | Reason |
|---------|--------|--------|
| Section 3.2 | Clarify notation | Rv1-C3 clarity concern |
### Wiki Updates
| Page | Update | Reason |
|------|--------|--------|
| claims/{slug} | Add open question | Rv2-C1 evidence gap |
### Suggested Experiments
| Experiment | Target Claim | Suggested by |
|-----------|-------------|--------------|
| ablation-dataset-x | [[claim-slug]] | Rv1-C2 |
→ Run `/exp-design ablation-dataset-x` to design follow-up
## Review LLM Stress-Test Summary
- Average score: {N}/5
- Scores 4-5: {N}/{total}
- Scores 1-3: {N}/{total} (all revised)
## Safety Checklist
- [x] No fabrication: all cited data exists in wiki/experiments
- [x] No overpromise: all committed experiments are specific and feasible
- [x] Full coverage: {N}/{N} concerns addressed (no omissions)
- [x] Challenged claims not presented as supported
6c. Final safety check:
- Full coverage: confirm every concern has a response (no omissions)
- No fabrication: every cited data point is recorded in wiki/experiments/ (traceable)
- No overpromise: supplementary experiment commitments are specific and feasible
- Honesty on weak claims: if claim confidence < 0.4, do not pretend evidence is sufficient
6d. Update wiki:
- For claims with evidence gaps: append reviewer-identified gaps to
## Open questionsinwiki/claims/{slug}.md - Append log:
python3 tools/research_wiki.py log wiki/ \ "rebuttal | {N} concerns addressed | {M} evidence gaps | stress-test avg: {score}/5"
Constraints
- No fabrication: never fabricate experiment data or results. Every cited number must be traceable to wiki/experiments/ with source annotated
- No overpromise: only commit to specific executable supplementary experiments. Use "we will run ablation on X with setup Y" not "we will investigate"
- Full coverage: every reviewer concern (Rvx-Cy) must have a response; omissions block output
- Evidence traceability: every piece of evidence cited in a response must be traceable to a wiki page with source slug annotated
- Do not directly modify wiki claims: rebuttal only appends suggestions to claims' Open questions; do not modify confidence/status
- Review LLM independence: during stress-test, follow cross-model-review.md; do not reveal response strategy to Review LLM
- Concern ID format: strictly use Rvx-Cy format (Rv1-C1, Rv1-C2, Rv2-C1) to ensure traceability
- Specific commitments: all revision commitments and experiment plans must be specific (specific Section, specific dataset, explicit metric)
- Output to wiki/outputs/: rebuttal files are stored uniformly in the wiki/outputs/ directory
Error Handling
- Review file not found: report error, list available files under raw/reviews/
- Review format cannot be parsed: fall back to plain-text processing; use LLM to extract concerns; annotate in report
- Concern cannot be mapped to a claim (unmapped): annotate as "unmapped"; still respond (based on paper content rather than wiki claim)
- Review LLM stress-test unavailable: skip Step 5; annotate in report "stress-test skipped: Review LLM unavailable"
- Evidence severely insufficient: if >50% of concerns have insufficient evidence, warn the user and suggest supplementing experiments first
- Wiki empty: warn that wiki knowledge base is empty; suggest running /ingest to populate claims and experiments
- All responses scored 1-2 by Review LLM: halt output, report requires re-analysis, suggest supplementing experiments first
Dependencies
Tools(via Bash)
python3 tools/research_wiki.py slug "{title}"— generate rebuttal slugpython3 tools/research_wiki.py log wiki/ "<message>"— append log entry
MCP Servers
mcp__llm-review__chat— Step 5 stress-test first roundmcp__llm-review__chat-reply— Step 5 stress-test subsequent rounds
Claude Code Native
Read— read review comments, wiki pages, shared referencesWrite— write rebuttal-{slug}.md, rebuttal-{slug}.txtGlob— find claims, experimentsGrep— search wiki for concern keywords
Shared References
.claude/skills/shared-references/cross-model-review.md— Review LLM stress-test independence principle
Suggested follow-up skills
/exp-design— design supplementary experiments for concerns with insufficient evidence/paper-draft— prepare revised paper (based on Paper Edits checklist)
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| finishing-a-development-branchobra/superpowersUse when implementation is complete, all tests pass, and you need to decide how to integrate the work | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |
| systematic-debuggingobra/superpowersUse when encountering any bug, test failure, or unexpected behavior, before proposing fixes | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |
| test-driven-developmentobra/superpowersUse when implementing any feature or bugfix, before writing implementation code | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |
| skill-creatoranthropics/skillsCreate new skills, modify and improve existing skills, and measure skill performance. Use when users want to create a skill from scratch, edit, or optimize an existing skill, run evals to test a skill, benchmark skill performance with variance analysis, or optimize a skill's description for better triggering accuracy. | Stars: 198k | Updated | Claude CodeCodex | Verified | Install |