Ar9av/PaperOrchestra / skills/paper-orchestra
paper-orchestra
VerifiedOrchestrate the full PaperOrchestra (Song et al., 2026, arXiv:2604.05018) five-agent pipeline to turn unstructured research materials (idea, experimental log, LaTeX template, conference guidelines, optional figures) into a submission-ready LaTeX manuscript and compiled PDF. TRIGGER when the user asks to "write a paper from my experiments", "turn this idea and these results into a paper", "generate a conference submission", "run paper-orchestra on X", or otherwise wants the end-to-end paper-writing pipeline. Coordinates the outline-agent, plotting-agent, literature-review-agent, section-writing-agent, and content-refinement-agent skills.
| File | Kind | Size |
|---|---|---|
| SKILL.md | skill md | 13.8 KB |
| references/anti-leakage-prompt.md | markdown | 2.5 KB |
| references/host-integration.md | markdown | 7.8 KB |
| references/io-contract.md | markdown | 7.1 KB |
| references/outline-reconciliation.md | markdown | 4.7 KB |
| references/paper-summary.md | markdown | 4.1 KB |
| references/pipeline.md | markdown | 7.2 KB |
| scripts/anti_leakage_check.py | script | 3.3 KB |
| scripts/build_pdf.py | script | 28.3 KB |
| scripts/check_idea_density.py | script | 5.6 KB |
| scripts/check_tex_packages.py | script | 5.9 KB |
| scripts/claim_evidence_gate.py | script | 16.1 KB |
| scripts/diff_outlines.py | script | 3.5 KB |
| scripts/init_workspace.py | script | 4.5 KB |
| scripts/test_claim_evidence_gate.py | script | 5.1 KB |
| scripts/validate_consistency.py | script | 9.7 KB |
| scripts/validate_inputs.py | script | 4.8 KB |
references/paper-summary.md
View on GitHub# PaperOrchestra — 1-page distillation
**Source:** Song, Y., Song, Y., Pfister, T., Yoon, J. *PaperOrchestra: A
Multi-Agent Framework for Automated AI Research Paper Writing.*
arXiv:2604.05018, 2026. <https://arxiv.org/pdf/2604.05018>
## Problem
Existing autonomous research-paper writers are tightly coupled to specific
experimental loops. They cannot transform unstructured human-provided
materials (an idea, an experimental log, a template) into a submission-ready
manuscript. Survey-only frameworks (AutoSurvey2, LiRA) lack full-paper
synthesis. Single-agent baselines hallucinate citations and produce shallow
literature reviews.
## Approach
A five-agent pipeline that maps `W(I, E, T, G, F) → P = (paper.tex, paper.pdf)`,
where:
- `I` = Idea Summary (Sparse or Dense markdown)
- `E` = Experimental Log (raw data, ablations, observations)
- `T` = LaTeX template
- `G` = Conference guidelines
- `F` = Optional pre-existing figures
The five agents:
1. **Outline Agent (1 LLM call)** — synthesizes the inputs into a JSON outline
with three sub-plans: a visualization plan, a literature search strategy
(macro for Intro + micro for Related Work), and a section-level writing
plan with mandatory citation hints for every dataset / metric / baseline.
2. **Plotting Agent (~20–30 calls)** — executes the visualization plan, using
PaperBanana-style few-shot retrieval and a VLM critique loop to
iteratively refine generated figures.
3. **Literature Review Agent (~20–30 calls)** — runs parallel candidate
discovery (10 web search workers), then sequential Semantic Scholar
verification (1 QPS, Levenshtein > 70 fuzzy title match), dedupes by
Semantic Scholar paper ID, and drafts Introduction + Related Work using
≥90% of the verified pool.
4. **Section Writing Agent (1 single multimodal call)** — drafts Abstract,
Methodology, Experiments, Conclusion; extracts numeric values from `E`
into LaTeX booktabs tables; integrates figures from Step 2.
5. **Content Refinement Agent (~5–7 calls)** — runs an AgentReview-style
simulated peer review loop; accepts revisions only if overall score
improves OR ties with non-negative net sub-axis change; reverts on
decrease or negative tie-break; halts at iteration cap (~3).
Steps 2 and 3 run in parallel.
## Engineering details that matter for fidelity
- **Universal Anti-Leakage Prompt** (App. D.4) prepended to every writing call.
- **Citation cutoff**: research_cutoff aligned to venue submission deadline
(Nov 2024 for CVPR 2025, Oct 2024 for ICLR 2025). Months default to first
day of the month for strict-predates comparison.
- **Citation verification**: Levenshtein title ratio > 70, year alignment
bonus, must have abstract, must strictly predate cutoff, dedup by S2 paperId.
- **Citation density rule**: ≥90% of the gathered pool MUST be cited in
Intro + Related Work.
- **Refinement safety**: agent must ignore reviewer requests for new
experiments and must never explicitly write "limitation" — both prevent
reward hacking against the simulated evaluator.
- **One single Section Writing call** — not chunked per section.
## Results (paper §5)
- Side-by-side LLM evaluation: PaperOrchestra wins 88–99% on literature
review quality and 39–86% on overall quality vs single-agent and
AI-Scientist-v2 baselines.
- Simulated acceptance rate (ScholarPeer): 84% (CVPR), 81% (ICLR) — close to
human GT rates of 86%/94%.
- Citation P1 Recall: 12.59–13.75% absolute over the strongest baseline.
- Refinement loop alone: +19% (CVPR) and +22% (ICLR) acceptance rate gains.
## What this repo implements
This repo is a **host-agent-pluggable skill pack** that lets any coding agent
(Claude Code, Cursor, Antigravity, Cline, Aider, OpenCode) execute the
PaperOrchestra pipeline. There are no API keys, no LLM SDKs, no embedded
network clients. Each skill is a markdown instruction document plus
deterministic local helper scripts. The host agent does all LLM reasoning,
web search, and Semantic Scholar fetching using its own native tools.
See `pipeline.md`, `io-contract.md`, and `host-integration.md` for details.
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| writing-skillsobra/superpowersUse when creating new skills, editing existing skills, or verifying skills work before deployment | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |
| xlsxanthropics/skillsUse this skill any time a spreadsheet file is the primary input or output. This means any task where the user wants to: open, read, edit, or fix an existing .xlsx, .xlsm, .xltx, .csv, or .tsv file (e.g., adding columns, computing formulas, formatting, charting, cleaning messy data); create a new spreadsheet from scratch or from other data sources; or convert between tabular file formats. Trigger especially when the user references a spreadsheet file by name or path — even casually (like "the xlsx in my downloads") — and wants something done to it or produced from it. Also trigger for cleaning or restructuring messy tabular data files (malformed rows, misplaced headers, junk data) into proper spreadsheets. The deliverable must be a spreadsheet file. Do NOT trigger when the primary deliverable is a Word document, HTML report, standalone Python script, database pipeline, or Google Sheets API integration, even if tabular data is involved. | Stars: 198k | Updated | Claude CodeCodex | Verified | Install |
| shipping-and-launchaddyosmani/agent-skillsPrepares production launches. Use when preparing to deploy to production, or when asking what needs to be in place before shipping. Use when you need a pre-launch checklist, when setting up monitoring, when planning a staged rollout, or when you need a rollback strategy. | Stars: 119k | Updated | Claude CodeCodex | Verified | Install |
| childhoodthedotmack/claude-memBabysit a shipped release after merge, publication, or deployment. Verify published packages, production behavior, installed plugins, and late CI or user reports until the observation window is clean; use babysit for PR readiness before release. | Stars: 118k | Updated | Claude CodeCodex | Verified | Install |