Skip to content

Ar9av/PaperOrchestra / skills/paper-autoraters

paper-autoraters

Verified

Run the four paper-quality autoraters from PaperOrchestra (arXiv:2604.05018, App. F.3) — Citation F1 (P0/P1 partition + Precision/Recall/F1), Literature Review Quality (6-axis 0-100 with anti-inflation rules), SxS Overall Paper Quality (side-by-side), and SxS Literature Review Quality (side-by-side). TRIGGER when the user asks to "score this paper draft", "evaluate against the benchmark", "compare two papers", or "run the autoraters".

references/citation-f1-prompt.md

View on GitHub
# Citation F1 — P0/P1 Partition prompt

**Source: arXiv:2604.05018, Appendix F.3, page 58 (verbatim).**

Use this as your system message to partition a paper's reference list into
P0 (must-cite) and P1 (good-to-cite) categories. Run it independently on
both the ground-truth paper and the generated paper, then feed both
partitions into `scripts/compute_f1.py` along with the resolved
Semantic Scholar IDs to compute Precision / Recall / F1.

---

```
You are an expert academic reviewer. Read the following paper text and
analyze its references.
Your goal is to categorize the provided references into two priorities:

Priority Levels

  - P0 (Must-Cite): Core citations strictly necessary for the paper. These
    MUST include:
      - Baselines directly compared against in experiments
      - Datasets the paper utilizes or evaluates on
      - Core methods the paper is directly building upon or modifying
      - Metrics or standard numbers heavily relied upon and cited from
        another paper

  - P1 (Good-To-Have): Supplemental citations. These include:
      - Standard background references covering broad history
      - General related work that is not directly competing or built-upon
      - Minor implementations or utility tools mentioned in passing

Paper Text:
{paper_text}

References List:
{references_str}

Output Format

Please return ONLY a JSON dictionary where the keys are the exact reference
numbers (e.g., "1", "2") and the values are either "P0" or "P1". Example
output:

```json
{{
    "1": "P0",
    "2": "P1",
    "3": "P0"
}}
```
```

---

## Substitution

| Placeholder | Source |
|---|---|
| `{paper_text}` | The full LaTeX or markdown text of the paper |
| `{references_str}` | The numbered reference list (extracted from `\bibliography{...}` or the References section) |

The model returns a JSON dict; the host agent saves it as
`gt_partition.json` (for the ground-truth paper) or `gen_partition.json`
(for the generated paper).

## How F1 is computed

After both partitions exist, the host agent must resolve every numbered
reference to a unique Semantic Scholar paper ID (using the same fuzzy
match + S2 verification logic as `literature-review-agent/scripts/`).
Then:

```
P0_GT  = set of S2 IDs from gt refs flagged P0
P0_Gen = set of S2 IDs from gen refs flagged P0
P0_Precision = |P0_GT ∩ P0_Gen| / |P0_Gen|
P0_Recall    = |P0_GT ∩ P0_Gen| / |P0_GT|
P0_F1        = 2 * P / R / (P + R)
```

Same for P1. Overall F1 uses the union of P0 and P1.

The deterministic computation lives in `scripts/compute_f1.py`.
add-product-e2e-evalpaperclipai/paperclipAdd or extend a Paperclip full-stack runner E2E workflow, fixture, matcher, or report evidence path for local or Daytona execution.Stars: 115kUpdated Claude CodeCodexVerifiedInstall
add-runner-evalpaperclipai/paperclipAdd or extend a Paperclip Runner protocol evaluation definition, roster, assertion, or report fixture with provenance and narrow validation.Stars: 115kUpdated Claude CodeCodexVerifiedInstall
diagnose-why-work-stoppedpaperclipai/paperclipCompatibility entry point for failed, stalled, looping, or over-recovered Paperclip work. Use fix-failed-paperclip-run for the full diagnosis, regression, repair, real verification, and green-PR workflow; honor an explicitly requested diagnosis-only scope.Stars: 115kUpdated Claude CodeCodexVerifiedInstall
fix-failed-paperclip-runpaperclipai/paperclipDiagnose and fix failed, stuck, slow, or incorrectly blocked Paperclip tasks and runs. Use with a task/run URL or failure report to trace the root cause, prove a regression red then green, check second-order effects, verify the real workflow, and deliver a PR with passing checks.Stars: 115kUpdated Claude CodeCodexVerifiedInstall