Ar9av/PaperOrchestra / skills/paper-autoraters
paper-autoraters
VerifiedRun the four paper-quality autoraters from PaperOrchestra (arXiv:2604.05018, App. F.3) — Citation F1 (P0/P1 partition + Precision/Recall/F1), Literature Review Quality (6-axis 0-100 with anti-inflation rules), SxS Overall Paper Quality (side-by-side), and SxS Literature Review Quality (side-by-side). TRIGGER when the user asks to "score this paper draft", "evaluate against the benchmark", "compare two papers", or "run the autoraters".
| File | Kind | Size |
|---|---|---|
| SKILL.md | skill md | 6.4 KB |
| references/citation-f1-prompt.md | markdown | 2.5 KB |
| references/litreview-quality-prompt.md | markdown | 7.9 KB |
| references/sxs-litreview-prompt.md | markdown | 2.3 KB |
| references/sxs-paper-quality-prompt.md | markdown | 3.4 KB |
| scripts/compute_f1.py | script | 3.6 KB |
references/sxs-litreview-prompt.md
View on GitHub# SxS Literature Review Quality Autorater — verbatim prompt
**Source: arXiv:2604.05018, Appendix F.3, pages 64–65 (verbatim).**
Use this as your system message to perform a side-by-side preference
comparison of just the literature review (Introduction + Related Work)
sections of two papers. Run twice with order swapped to mitigate
positional bias.
---
```
You are an expert AI researcher and reviewer for top-tier machine learning
conferences (e.g., CVPR, NeurIPS, ICLR).
Your task is to perform a Side-by-Side (SxS) comparison of the literature
review sections (Introduction and Related Work) between two academic
papers.
The ordering of the papers is arbitrary and does not indicate quality.
Evaluate each paper independently before comparing them.
Do not base your decision solely on length or verbosity.
Critical Evaluation Criteria
1. Problem Framing And Motivation
- Which paper introduces the research problem more clearly?
- Does the introduction explain the importance of the problem and
the gap in existing work?
2. Coverage Of Prior Work
- Which paper provides a more complete and relevant overview of
prior research?
3. Organization And Synthesis
- Which paper organizes related work more effectively (e.g.,
grouping by themes or approaches)?
- Does it synthesize prior work rather than simply listing papers?
4. Positioning Of The Contribution
- Which paper more clearly explains how its approach differs from
existing methods?
5. Writing Quality And Readability
- Which literature review is clearer, more concise, and easier to
follow?
Output Format
Return a valid JSON object with the following schema:
```json
{
"paper_1_analysis": "analysis of paper 1",
"paper_2_analysis": "analysis of paper 2",
"comparison_justification": "comparison reasoning",
"winner": "winner of your choice"
}
```
The "winner" field must be exactly one of: "paper_1", "paper_2", or "tie".
```
---
## Inputs
This autorater needs only the **Introduction and Related Work** sections of
each paper, NOT the full document. The host agent should:
1. Extract Intro + Related Work from both papers (LaTeX section commands or
PDF section detection).
2. Pass them as `paper_1` and `paper_2`.
3. Run twice, swapping order, to mitigate positional bias (see
`sxs-paper-quality-prompt.md` for the protocol).
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| add-product-e2e-evalpaperclipai/paperclipAdd or extend a Paperclip full-stack runner E2E workflow, fixture, matcher, or report evidence path for local or Daytona execution. | Stars: 115k | Updated | Claude CodeCodex | Verified | Install |
| add-runner-evalpaperclipai/paperclipAdd or extend a Paperclip Runner protocol evaluation definition, roster, assertion, or report fixture with provenance and narrow validation. | Stars: 115k | Updated | Claude CodeCodex | Verified | Install |
| diagnose-why-work-stoppedpaperclipai/paperclipCompatibility entry point for failed, stalled, looping, or over-recovered Paperclip work. Use fix-failed-paperclip-run for the full diagnosis, regression, repair, real verification, and green-PR workflow; honor an explicitly requested diagnosis-only scope. | Stars: 115k | Updated | Claude CodeCodex | Verified | Install |
| fix-failed-paperclip-runpaperclipai/paperclipDiagnose and fix failed, stuck, slow, or incorrectly blocked Paperclip tasks and runs. Use with a task/run URL or failure report to trace the root cause, prove a regression red then green, check second-order effects, verify the real workflow, and deliver a PR with passing checks. | Stars: 115k | Updated | Claude CodeCodex | Verified | Install |