Skip to content

Ar9av/PaperOrchestra / skills/paper-autoraters

paper-autoraters

Verified

Run the four paper-quality autoraters from PaperOrchestra (arXiv:2604.05018, App. F.3) — Citation F1 (P0/P1 partition + Precision/Recall/F1), Literature Review Quality (6-axis 0-100 with anti-inflation rules), SxS Overall Paper Quality (side-by-side), and SxS Literature Review Quality (side-by-side). TRIGGER when the user asks to "score this paper draft", "evaluate against the benchmark", "compare two papers", or "run the autoraters".

references/sxs-paper-quality-prompt.md

View on GitHub
# SxS Overall Paper Quality Autorater — verbatim prompt

**Source: arXiv:2604.05018, Appendix F.3, pages 63–64 (verbatim).**

Use this as your system message to perform a side-by-side preference
comparison between two paper drafts. To mitigate positional bias, run the
comparison TWICE with the paper order swapped, then aggregate.

---

```
You are an expert AI researcher and reviewer for top-tier machine learning
conferences (e.g., CVPR, NeurIPS, ICLR).
Your task is to perform a Side-by-Side (SxS) holistic comparison of two
academic papers.
The two papers describe the same or highly similar research ideas. Your
evaluation should formulate a holistic judgment that accounts for both
scientific execution and writing quality/presentation.

The ordering of the papers is arbitrary and does not indicate quality.
Evaluate each paper independently before comparing them.
Do not base your decision solely on length or verbosity.

Critical Evaluation Criteria

  1. Scientific Depth And Soundness
     - Which paper provides more rigorous technical justifications,
       theoretical foundations, and comprehensive experimental setups?

  2. Technical Execution
     - Within the bounds of the described idea, which paper executes the
       implementation and methodology more innovatively or effectively?

  3. Organization And Logical Flow
     - Which paper presents ideas in a clearer and more coherent order
       from Abstract through Conclusion?
     - Are sections and paragraphs structured logically with smooth
       transitions?

  4. Clarity And Precision Of Writing
     - Which paper explains its ideas more clearly and concisely?
     - Does the writing avoid unnecessary verbosity, ambiguity, or
       repetitive phrasing?

  5. Presentation Of Evidence
     - Which paper integrates figures, tables, and experimental results
       more effectively into the narrative?
     - Are visuals clearly referenced and explained in the text?

  6. Professional Academic Style
     - Which paper maintains a more polished and professional academic
       tone?
     - Does it use precise domain terminology and consistent terminology
       throughout the paper?

Output Format

Return a valid JSON object with the following schema:

```json
{
  "paper_1_holistic_analysis":
    "analysis of paper_1 writing, presentation, and scientific execution",
  "paper_2_holistic_analysis":
    "analysis of paper_2 writing, presentation, and scientific execution",
  "comparison_justification":
    "comparison reasoning",
  "winner":
    "winner of your choice"
}
```

The "winner" field must be exactly one of: "paper_1", "paper_2", or "tie".
```

---

## Positional bias mitigation protocol

The paper notes (§5.4): "human preferences correlate strongly with our
GPT-5 evaluator for Overall Quality (Pearson r = 0.6458, Spearman ρ =
0.6355). Literature review correlation is lower due to inherent LLM
self-bias." To get a robust SxS verdict, run the comparison twice:

```
Call 1: paper_A → paper_1,  paper_B → paper_2,  result1
Call 2: paper_B → paper_1,  paper_A → paper_2,  result2

normalize both results to "A wins" / "B wins" / "tie", then:

Final outcome:
  - WIN  for A:    A wins in both calls
  - LOSS for A:    B wins in both calls
  - TIE:           one win + one tie, or two ties, or A wins one + B wins one
```

The paper uses this exact protocol — see §5.2 "(2) SxS Paper Quality"
description.
add-product-e2e-evalpaperclipai/paperclipAdd or extend a Paperclip full-stack runner E2E workflow, fixture, matcher, or report evidence path for local or Daytona execution.Stars: 115kUpdated Claude CodeCodexVerifiedInstall
add-runner-evalpaperclipai/paperclipAdd or extend a Paperclip Runner protocol evaluation definition, roster, assertion, or report fixture with provenance and narrow validation.Stars: 115kUpdated Claude CodeCodexVerifiedInstall
diagnose-why-work-stoppedpaperclipai/paperclipCompatibility entry point for failed, stalled, looping, or over-recovered Paperclip work. Use fix-failed-paperclip-run for the full diagnosis, regression, repair, real verification, and green-PR workflow; honor an explicitly requested diagnosis-only scope.Stars: 115kUpdated Claude CodeCodexVerifiedInstall
fix-failed-paperclip-runpaperclipai/paperclipDiagnose and fix failed, stuck, slow, or incorrectly blocked Paperclip tasks and runs. Use with a task/run URL or failure report to trace the root cause, prove a regression red then green, check second-order effects, verify the real workflow, and deliver a PR with passing checks.Stars: 115kUpdated Claude CodeCodexVerifiedInstall