Ar9av/PaperOrchestra / skills/paper-writing-bench
paper-writing-bench
VerifiedReverse-engineer raw materials (Sparse idea, Dense idea, experimental log) from an existing AI research paper to build a benchmark case for evaluating paper-writing pipelines. Replicates the PaperWritingBench dataset construction procedure from arXiv:2604.05018 §3 / App. C. TRIGGER when the user asks to "build a benchmark case from this paper", "reverse-engineer raw materials", or "evaluate my pipeline against PaperWritingBench".
| File | Kind | Size |
|---|---|---|
| SKILL.md | skill md | 5.5 KB |
| references/bench-overview.md | markdown | 4.2 KB |
| references/dense-idea-prompt.md | markdown | 3.2 KB |
| references/experimental-log-prompt.md | markdown | 3.2 KB |
| references/sparse-idea-prompt.md | markdown | 2.7 KB |
references/bench-overview.md
View on GitHub# PaperWritingBench Overview
Source: arXiv:2604.05018, §3 ("Task and Dataset"), App. C ("Dataset Details").
## Composition
| Venue | Papers | Cutoff | Avg figs | Avg tables | Avg cites |
|---|---|---|---|---|---|
| CVPR 2025 | 100 | 2024-11-01 | 5.20 ± 1.73 | 4.20 ± 1.65 | 58.52 ± 17.55 |
| ICLR 2025 | 100 | 2024-10-01 | 9.19 ± 5.39 | 8.13 ± 5.19 | 59.18 ± 20.01 |
ICLR 2025 papers exhibit "higher visual and analytical density than CVPR 2025"
(App. C.1) — roughly twice as many figures and tables, and substantially
longer experimental logs (mean 2,387 vs 1,530 words).
## Idea variant statistics
| Variant | Avg word count CVPR | Avg word count ICLR |
|---|---|---|
| Dense | 1,082.82 ± 263.45 | 1,056.78 ± 165.62 |
| Sparse | 591.42 ± 52.68 | 586.96 ± 53.81 |
Sparse ideas are roughly half the length of Dense ideas. Both are kept
deliberately constrained — the bench tests whether the pipeline can
reconstruct a full paper from sparse human notes, not from a complete
draft.
## Construction pipeline
Per App. C.2:
1. **Sample** papers from OpenReview (ICLR) and the CVF Open Access
repository (CVPR).
2. **Extract** with MinerU (Wang et al., 2024) for the markdown body and
PDFFigures 2.0 (Clark and Divvala, 2016) for visual entities and
captions. Discard incomplete or misparsed samples.
3. **Reverse-engineer** raw materials with Gemini-3.1-Pro (or any capable
LLM in your host) using the three verbatim prompts in this skill's
`references/`:
- `sparse-idea-prompt.md`
- `dense-idea-prompt.md`
- `experimental-log-prompt.md`
4. **Anonymize**: strip authors, titles, citations, URLs, figure/table
references. The result must be self-contained.
5. **Inject visual context** (App. C.2 "Structured Context Injection"):
when generating the experimental log, the paper passes the actual
figure images alongside the text to the extraction LLM, so the model
can convert visual insights into standalone factual observations
(e.g., "training loss converged after 200 epochs"). This avoids the
common failure mode of "naive text-only extraction corrupts tabular
and mathematical data."
## Why two variants
The Dense vs Sparse split tests robustness to input granularity:
- **Dense**: closer to a real research note from a scientist — preserves
the math, the architectural specifics, the variable definitions. Tests
whether the system can faithfully translate detailed technical content
into a polished manuscript.
- **Sparse**: closer to a brainstorming note — high-level concept only.
Tests whether the system can fill in formal mathematical structure
from a rough sketch.
The paper's ablation (Table 5) shows Dense wins more on "Overall Paper
Quality" (43-56% Dense win vs 18-24% Sparse win) but the gap is much
smaller for "Lit Review" (32-40% Dense win vs 33-40% Sparse win) —
demonstrating that the literature search step is robust to input density.
## Cutoff dates
The paper aligns research cutoff with venue submission deadline:
| Venue | Submission deadline | Cutoff used |
|---|---|---|
| CVPR 2025 | November 2024 | `2024-11-01` |
| ICLR 2025 | October 2024 | `2024-10-01` |
For other venues, use one month before the stated submission deadline as
the cutoff. Encode as `YYYY-MM-DD`. Months default to day-1 (e.g., "October"
→ `2024-10-01`).
## Using the bench with your own pipeline
To benchmark a coding agent's paper-writing skill, run:
```
1. Pick a paper from your held-out set (NOT in the LLM's training data
ideally — the paper notes pre-training contamination is an inherent
risk; use unpublished or recent work to mitigate).
2. Run paper-writing-bench to extract idea_sparse.md / idea_dense.md /
experimental_log.md.
3. Run paper-orchestra on those inputs (with the venue's template and
guidelines).
4. Run paper-autoraters to compare the generated paper to the original.
```
The autoraters skill ships the four metrics from App. F.3:
- **Citation F1** (P0 must-cite, P1 good-to-cite) — checks the generated
bibliography against the ground truth.
- **Literature Review Quality** (6-axis 0-100) — scores Intro + Related Work.
- **SxS Overall Paper Quality** — full-paper side-by-side comparison.
- **SxS Literature Review Quality** — Intro+Related Work side-by-side.
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| writing-skillsobra/superpowersUse when creating new skills, editing existing skills, or verifying skills work before deployment | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |
| xlsxanthropics/skillsUse this skill any time a spreadsheet file is the primary input or output. This means any task where the user wants to: open, read, edit, or fix an existing .xlsx, .xlsm, .xltx, .csv, or .tsv file (e.g., adding columns, computing formulas, formatting, charting, cleaning messy data); create a new spreadsheet from scratch or from other data sources; or convert between tabular file formats. Trigger especially when the user references a spreadsheet file by name or path — even casually (like "the xlsx in my downloads") — and wants something done to it or produced from it. Also trigger for cleaning or restructuring messy tabular data files (malformed rows, misplaced headers, junk data) into proper spreadsheets. The deliverable must be a spreadsheet file. Do NOT trigger when the primary deliverable is a Word document, HTML report, standalone Python script, database pipeline, or Google Sheets API integration, even if tabular data is involved. | Stars: 198k | Updated | Claude CodeCodex | Verified | Install |
| shipping-and-launchaddyosmani/agent-skillsPrepares production launches. Use when preparing to deploy to production, or when asking what needs to be in place before shipping. Use when you need a pre-launch checklist, when setting up monitoring, when planning a staged rollout, or when you need a rollback strategy. | Stars: 119k | Updated | Claude CodeCodex | Verified | Install |
| childhoodthedotmack/claude-memBabysit a shipped release after merge, publication, or deployment. Verify published packages, production behavior, installed plugins, and late CI or user reports until the observation window is clean; use babysit for PR readiness before release. | Stars: 118k | Updated | Claude CodeCodex | Verified | Install |