Skip to content

Ar9av/PaperOrchestra / skills/paper-autoraters

paper-autoraters

Verified

Run the four paper-quality autoraters from PaperOrchestra (arXiv:2604.05018, App. F.3) — Citation F1 (P0/P1 partition + Precision/Recall/F1), Literature Review Quality (6-axis 0-100 with anti-inflation rules), SxS Overall Paper Quality (side-by-side), and SxS Literature Review Quality (side-by-side). TRIGGER when the user asks to "score this paper draft", "evaluate against the benchmark", "compare two papers", or "run the autoraters".

references/sxs-litreview-prompt.md

View on GitHub
# SxS Literature Review Quality Autorater — verbatim prompt

**Source: arXiv:2604.05018, Appendix F.3, pages 64–65 (verbatim).**

Use this as your system message to perform a side-by-side preference
comparison of just the literature review (Introduction + Related Work)
sections of two papers. Run twice with order swapped to mitigate
positional bias.

---

```
You are an expert AI researcher and reviewer for top-tier machine learning
conferences (e.g., CVPR, NeurIPS, ICLR).

Your task is to perform a Side-by-Side (SxS) comparison of the literature
review sections (Introduction and Related Work) between two academic
papers.

The ordering of the papers is arbitrary and does not indicate quality.
Evaluate each paper independently before comparing them.
Do not base your decision solely on length or verbosity.

Critical Evaluation Criteria

  1. Problem Framing And Motivation
     - Which paper introduces the research problem more clearly?
     - Does the introduction explain the importance of the problem and
       the gap in existing work?

  2. Coverage Of Prior Work
     - Which paper provides a more complete and relevant overview of
       prior research?

  3. Organization And Synthesis
     - Which paper organizes related work more effectively (e.g.,
       grouping by themes or approaches)?
     - Does it synthesize prior work rather than simply listing papers?

  4. Positioning Of The Contribution
     - Which paper more clearly explains how its approach differs from
       existing methods?

  5. Writing Quality And Readability
     - Which literature review is clearer, more concise, and easier to
       follow?

Output Format

Return a valid JSON object with the following schema:

```json
{
  "paper_1_analysis": "analysis of paper 1",
  "paper_2_analysis": "analysis of paper 2",
  "comparison_justification": "comparison reasoning",
  "winner": "winner of your choice"
}
```

The "winner" field must be exactly one of: "paper_1", "paper_2", or "tie".
```

---

## Inputs

This autorater needs only the **Introduction and Related Work** sections of
each paper, NOT the full document. The host agent should:

1. Extract Intro + Related Work from both papers (LaTeX section commands or
   PDF section detection).
2. Pass them as `paper_1` and `paper_2`.
3. Run twice, swapping order, to mitigate positional bias (see
   `sxs-paper-quality-prompt.md` for the protocol).
add-product-e2e-evalpaperclipai/paperclipAdd or extend a Paperclip full-stack runner E2E workflow, fixture, matcher, or report evidence path for local or Daytona execution.Stars: 115kUpdated Claude CodeCodexVerifiedInstall
add-runner-evalpaperclipai/paperclipAdd or extend a Paperclip Runner protocol evaluation definition, roster, assertion, or report fixture with provenance and narrow validation.Stars: 115kUpdated Claude CodeCodexVerifiedInstall
diagnose-why-work-stoppedpaperclipai/paperclipCompatibility entry point for failed, stalled, looping, or over-recovered Paperclip work. Use fix-failed-paperclip-run for the full diagnosis, regression, repair, real verification, and green-PR workflow; honor an explicitly requested diagnosis-only scope.Stars: 115kUpdated Claude CodeCodexVerifiedInstall
fix-failed-paperclip-runpaperclipai/paperclipDiagnose and fix failed, stuck, slow, or incorrectly blocked Paperclip tasks and runs. Use with a task/run URL or failure report to trace the root cause, prove a regression red then green, check second-order effects, verify the real workflow, and deliver a PR with passing checks.Stars: 115kUpdated Claude CodeCodexVerifiedInstall