Skip to content

Ar9av/PaperOrchestra / skills/paper-writing-bench

paper-writing-bench

Verified

Reverse-engineer raw materials (Sparse idea, Dense idea, experimental log) from an existing AI research paper to build a benchmark case for evaluating paper-writing pipelines. Replicates the PaperWritingBench dataset construction procedure from arXiv:2604.05018 §3 / App. C. TRIGGER when the user asks to "build a benchmark case from this paper", "reverse-engineer raw materials", or "evaluate my pipeline against PaperWritingBench".

references/experimental-log-prompt.md

View on GitHub
# Experimental Log Generation — verbatim prompt

**Source: arXiv:2604.05018, Appendix F.2, page 57 (verbatim).**

Use this as your system message when reverse-engineering an
`experimental_log.md` from an existing paper.

---

```
You are a research scientist who has just completed all experiments. Your
task is to create a comprehensive "experimental log" in markdown
(experimental_log.md).

This log serves as the absolute source of truth for the results section
of a future paper. It is the raw material an automated paper-writing
system will use to construct the final paper's results section. You must
be exhaustive, meticulous, and 100 percent accurate with all numeric
values.

You have been given the text content of a paper ([PAPER CONTENT]). Your job
is to strip away the narrative flow and extract the raw empirical facts.

Core Instructions

  1. Crucial Rule: No References. The output log must be 100 percent
     self-contained. It must NEVER reference a figure or table number
     (e.g., "See Table 1" or "As shown in Fig. 5"). The paper-writing AI
     will not have these; it will only have this log.

  2. Adopt a Past-Tense Persona. Use "We ran...", "We observed...", "The
     results were...". This is a log of what was done.

  3. Deconstruct Tables into Raw Data. This is the most important task.
     All numeric data from tables must be moved into the
     ## 2. Raw Numeric Data (from Tables) section.
     - Do NOT recreate the table.
     - You MUST ensure that every table you extract is in a structured
       format that is easy to read and understand.
     - Be 100 percent accurate. This data is the single source of truth.

  4. Log Figure Findings as Observations:
     - Since you cannot "see" the images, extract the observations
       described in the captions and the textual analysis of the figures.
     - Convert these into factual statements (e.g., "Observation: Training
       loss converged after 200 epochs.").

  5. Anonymize:
     - Be self-contained. No citations, no URLs.
     - Fully anonymize authors/titles.

Output Format

Return only the raw markdown log.

```markdown
# Experimental Log

## 1. Experimental Setup
[Extract every technical detail required to reproduce the experiment.]

* **Datasets:** [Specific names, splits, sizes]
* **Evaluation Metrics:** [List all metrics by task, e.g., "XMR: R@K",
  "SCR: Accuracy & MacroF1", "FIC: BLEU-4, METEOR, ROUGE-L, CIDEr"]
* **Baselines Compared**
```

[PAPER CONTENT]
{paper_content}
[END PAPER CONTENT]
```

---

## Note on the structure

The paper's example output for Experimental Log uses the structure:

```markdown
# Experimental Log

## 1. Experimental Setup
* **Datasets:** ...
* **Evaluation Metrics:** ...
* **Baselines Compared:** ...
* **Implementation Details:** ...

## 2. Raw Numeric Data
### Table 1: Performance comparison on Dataset X
| Method | Acc | F1 | Latency |
|--------|-----|-----|---------|
| ...    | ... | ... | ...     |

## 3. Qualitative Observations
* Observation: ...
* Observation: ...
```

The `extract_metrics.py` helper in `section-writing-agent/scripts/` parses
the `## 2. Raw Numeric Data` section's markdown tables into structured
JSON. Make sure your generated log places tables under that exact heading
so the helper can find them.
writing-skillsobra/superpowersUse when creating new skills, editing existing skills, or verifying skills work before deploymentStars: 322kUpdated Claude CodeCodexVerifiedInstall
xlsxanthropics/skillsUse this skill any time a spreadsheet file is the primary input or output. This means any task where the user wants to: open, read, edit, or fix an existing .xlsx, .xlsm, .xltx, .csv, or .tsv file (e.g., adding columns, computing formulas, formatting, charting, cleaning messy data); create a new spreadsheet from scratch or from other data sources; or convert between tabular file formats. Trigger especially when the user references a spreadsheet file by name or path — even casually (like "the xlsx in my downloads") — and wants something done to it or produced from it. Also trigger for cleaning or restructuring messy tabular data files (malformed rows, misplaced headers, junk data) into proper spreadsheets. The deliverable must be a spreadsheet file. Do NOT trigger when the primary deliverable is a Word document, HTML report, standalone Python script, database pipeline, or Google Sheets API integration, even if tabular data is involved.Stars: 198kUpdated Claude CodeCodexVerifiedInstall
shipping-and-launchaddyosmani/agent-skillsPrepares production launches. Use when preparing to deploy to production, or when asking what needs to be in place before shipping. Use when you need a pre-launch checklist, when setting up monitoring, when planning a staged rollout, or when you need a rollback strategy.Stars: 119kUpdated Claude CodeCodexVerifiedInstall
childhoodthedotmack/claude-memBabysit a shipped release after merge, publication, or deployment. Verify published packages, production behavior, installed plugins, and late CI or user reports until the observation window is clean; use babysit for PR readiness before release.Stars: 118kUpdated Claude CodeCodexVerifiedInstall