Ar9av/PaperOrchestra / skills/paper-autoraters
paper-autoraters
VerifiedRun the four paper-quality autoraters from PaperOrchestra (arXiv:2604.05018, App. F.3) — Citation F1 (P0/P1 partition + Precision/Recall/F1), Literature Review Quality (6-axis 0-100 with anti-inflation rules), SxS Overall Paper Quality (side-by-side), and SxS Literature Review Quality (side-by-side). TRIGGER when the user asks to "score this paper draft", "evaluate against the benchmark", "compare two papers", or "run the autoraters".
| File | Kind | Size |
|---|---|---|
| SKILL.md | skill md | 6.4 KB |
| references/citation-f1-prompt.md | markdown | 2.5 KB |
| references/litreview-quality-prompt.md | markdown | 7.9 KB |
| references/sxs-litreview-prompt.md | markdown | 2.3 KB |
| references/sxs-paper-quality-prompt.md | markdown | 3.4 KB |
| scripts/compute_f1.py | script | 3.6 KB |
references/sxs-paper-quality-prompt.md
View on GitHub# SxS Overall Paper Quality Autorater — verbatim prompt
**Source: arXiv:2604.05018, Appendix F.3, pages 63–64 (verbatim).**
Use this as your system message to perform a side-by-side preference
comparison between two paper drafts. To mitigate positional bias, run the
comparison TWICE with the paper order swapped, then aggregate.
---
```
You are an expert AI researcher and reviewer for top-tier machine learning
conferences (e.g., CVPR, NeurIPS, ICLR).
Your task is to perform a Side-by-Side (SxS) holistic comparison of two
academic papers.
The two papers describe the same or highly similar research ideas. Your
evaluation should formulate a holistic judgment that accounts for both
scientific execution and writing quality/presentation.
The ordering of the papers is arbitrary and does not indicate quality.
Evaluate each paper independently before comparing them.
Do not base your decision solely on length or verbosity.
Critical Evaluation Criteria
1. Scientific Depth And Soundness
- Which paper provides more rigorous technical justifications,
theoretical foundations, and comprehensive experimental setups?
2. Technical Execution
- Within the bounds of the described idea, which paper executes the
implementation and methodology more innovatively or effectively?
3. Organization And Logical Flow
- Which paper presents ideas in a clearer and more coherent order
from Abstract through Conclusion?
- Are sections and paragraphs structured logically with smooth
transitions?
4. Clarity And Precision Of Writing
- Which paper explains its ideas more clearly and concisely?
- Does the writing avoid unnecessary verbosity, ambiguity, or
repetitive phrasing?
5. Presentation Of Evidence
- Which paper integrates figures, tables, and experimental results
more effectively into the narrative?
- Are visuals clearly referenced and explained in the text?
6. Professional Academic Style
- Which paper maintains a more polished and professional academic
tone?
- Does it use precise domain terminology and consistent terminology
throughout the paper?
Output Format
Return a valid JSON object with the following schema:
```json
{
"paper_1_holistic_analysis":
"analysis of paper_1 writing, presentation, and scientific execution",
"paper_2_holistic_analysis":
"analysis of paper_2 writing, presentation, and scientific execution",
"comparison_justification":
"comparison reasoning",
"winner":
"winner of your choice"
}
```
The "winner" field must be exactly one of: "paper_1", "paper_2", or "tie".
```
---
## Positional bias mitigation protocol
The paper notes (§5.4): "human preferences correlate strongly with our
GPT-5 evaluator for Overall Quality (Pearson r = 0.6458, Spearman ρ =
0.6355). Literature review correlation is lower due to inherent LLM
self-bias." To get a robust SxS verdict, run the comparison twice:
```
Call 1: paper_A → paper_1, paper_B → paper_2, result1
Call 2: paper_B → paper_1, paper_A → paper_2, result2
normalize both results to "A wins" / "B wins" / "tie", then:
Final outcome:
- WIN for A: A wins in both calls
- LOSS for A: B wins in both calls
- TIE: one win + one tie, or two ties, or A wins one + B wins one
```
The paper uses this exact protocol — see §5.2 "(2) SxS Paper Quality"
description.
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| add-product-e2e-evalpaperclipai/paperclipAdd or extend a Paperclip full-stack runner E2E workflow, fixture, matcher, or report evidence path for local or Daytona execution. | Stars: 115k | Updated | Claude CodeCodex | Verified | Install |
| add-runner-evalpaperclipai/paperclipAdd or extend a Paperclip Runner protocol evaluation definition, roster, assertion, or report fixture with provenance and narrow validation. | Stars: 115k | Updated | Claude CodeCodex | Verified | Install |
| diagnose-why-work-stoppedpaperclipai/paperclipCompatibility entry point for failed, stalled, looping, or over-recovered Paperclip work. Use fix-failed-paperclip-run for the full diagnosis, regression, repair, real verification, and green-PR workflow; honor an explicitly requested diagnosis-only scope. | Stars: 115k | Updated | Claude CodeCodex | Verified | Install |
| fix-failed-paperclip-runpaperclipai/paperclipDiagnose and fix failed, stuck, slow, or incorrectly blocked Paperclip tasks and runs. Use with a task/run URL or failure report to trace the root cause, prove a regression red then green, check second-order effects, verify the real workflow, and deliver a PR with passing checks. | Stars: 115k | Updated | Claude CodeCodex | Verified | Install |