Ar9av/PaperOrchestra / skills/content-refinement-agent
content-refinement-agent
VerifiedStep 5 of the PaperOrchestra pipeline (arXiv:2604.05018). Iteratively refine drafts/paper.tex by simulating peer review and applying targeted revisions, with strict accept/revert halt rules, deterministic 0-100 decision bands (Accept/Minor/Major/Reject) that drive a target-met early stop, and a Devil's Advocate concession-threshold guard that blocks acceptance on unresolved critical findings. Maintains a worklog and snapshots each iteration so revert is real, not symbolic. TRIGGER when the orchestrator delegates Step 5 or when the user asks to "refine the draft", "iterate on the paper", or "run peer review on this paper".
| File | Kind | Size |
|---|---|---|
| SKILL.md | skill md | 19.3 KB |
| references/ai-failure-modes.md | markdown | 813 B |
| references/claim-evidence-map.md | markdown | 3.1 KB |
| references/da-reviewer.md | markdown | 3.9 KB |
| references/halt-rules.md | markdown | 6.1 KB |
| references/prompt.md | markdown | 5.7 KB |
| references/reverse-outline.md | markdown | 3.0 KB |
| references/reviewer-rubric.md | markdown | 6.8 KB |
| references/safe-revision-rules.md | markdown | 4.8 KB |
| references/writing-quality-check.md | markdown | 720 B |
| scripts/apply_worklog.py | script | 3.1 KB |
| scripts/concession_guard.py | script | 7.5 KB |
| scripts/decision_band.py | script | 3.6 KB |
| scripts/reverse_outline.py | script | 10.3 KB |
| scripts/score_delta.py | script | 7.2 KB |
| scripts/score_trajectory.py | script | 8.6 KB |
| scripts/snapshot.py | script | 1.6 KB |
| scripts/test_reverse_outline.py | script | 5.7 KB |
| scripts/update_critique_memory.py | script | 11.1 KB |
references/da-reviewer.md
View on GitHub# Devil's Advocate Reviewer Protocol
## Role
One of the simulated peer reviewers is designated the Devil's Advocate (DA).
The DA's job is to challenge the paper's core claims from first principles,
not to find polish issues (those are other reviewers' job).
## DA Attack Targets (in priority order)
1. **Causal overclaiming** — Does the paper say "X causes Y" when it only shows
correlation?
2. **Ablation coverage** — Does every claimed component have an ablation? If not,
flag missing ablations.
3. **Baseline fairness** — Are baselines run with the same compute budget and
tuning effort?
4. **Generalization claims** — Does the paper claim broad applicability from narrow
experiments (e.g., 1 dataset)?
5. **Novelty inflation** — Is the "novel" contribution already present in cited
works?
## Concession Threshold
The DA must score each rebuttal from other reviewers 1–5 before updating its
position:
- Score 5: rebuttal directly addresses the attack with paper evidence
→ concession allowed
- Score 4: rebuttal provides strong indirect evidence → concession allowed
- Score 3: partial rebuttal → DA holds position, restates attack more specifically
- Score 1–2: weak rebuttal or no response → DA escalates (marks as CRITICAL if
unaddressed after all reviewers weigh in)
**IRON RULE:** No consecutive concessions. The DA may concede at most once per two
review rounds.
DA CRITICAL findings block the "refinement accepted" decision regardless of overall
rubric scores.
## What DA CRITICAL Means
If the DA issues a CRITICAL finding, `score_delta.py` exit code is overridden to
2 (REVERT). The revision must specifically address the CRITICAL finding before
continuing.
Log in worklog.json: `{da_critical: true, finding: "..."}`
## Deterministic enforcement: `scripts/concession_guard.py`
The concession threshold and the no-consecutive-concessions iron rule are easy
for a simulated reviewer to quietly relax — it caves. To make them
non-negotiable, record the DA's findings and concession decisions in a
**concession log** and run `concession_guard.py` each iteration. The script
re-derives which concessions are valid and whether any CRITICAL is still
standing; the host agent must obey its verdict over the LLM's prose.
Concession log schema (`workspace/refinement/da_concessions.json`):
```json
{
"rounds": [
{
"round": 1,
"findings": [
{
"id": "F1",
"severity": "critical",
"attack": "Sec 4 claims X *causes* Y from correlation only.",
"rebuttal_score": 2,
"conceded": false,
"resolved": false
}
]
}
]
}
```
- `rebuttal_score` (1–5) — the DA's score of the author/revision rebuttal,
using the concession-threshold scale above.
- `conceded` — did the DA drop the attack this round?
- `resolved` — was the underlying issue actually fixed in the revision?
```bash
python skills/content-refinement-agent/scripts/concession_guard.py \
--log workspace/refinement/da_concessions.json \
--out workspace/refinement/iter<N>/da_guard.json
```
Verdict → loop action:
| Guard exit | Meaning | Host action |
|---|---|---|
| 0 | CLEAR — no standing critical, no violations | accept may proceed |
| 1 | BLOCK — a critical is still standing | treat the iteration as **REVERT** (force `score_delta.py` outcome to exit 2) and require the next revision to address it |
| 2 | WARN — a concession was rejected (caving or consecutive) but no critical is blocked | the DA must restate the attack; do not let the rejected concession stand |
| 3 | input / schema error | fix the log |
The guard rejects (does not honor) any concession made at `rebuttal_score < 4`
or in a round immediately following another conceding round, and restores the
affected finding to "standing". A standing CRITICAL blocks acceptance
regardless of rubric scores — this is the deterministic backstop behind the
prose rules above.
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| writing-skillsobra/superpowersUse when creating new skills, editing existing skills, or verifying skills work before deployment | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |
| xlsxanthropics/skillsUse this skill any time a spreadsheet file is the primary input or output. This means any task where the user wants to: open, read, edit, or fix an existing .xlsx, .xlsm, .xltx, .csv, or .tsv file (e.g., adding columns, computing formulas, formatting, charting, cleaning messy data); create a new spreadsheet from scratch or from other data sources; or convert between tabular file formats. Trigger especially when the user references a spreadsheet file by name or path — even casually (like "the xlsx in my downloads") — and wants something done to it or produced from it. Also trigger for cleaning or restructuring messy tabular data files (malformed rows, misplaced headers, junk data) into proper spreadsheets. The deliverable must be a spreadsheet file. Do NOT trigger when the primary deliverable is a Word document, HTML report, standalone Python script, database pipeline, or Google Sheets API integration, even if tabular data is involved. | Stars: 198k | Updated | Claude CodeCodex | Verified | Install |
| shipping-and-launchaddyosmani/agent-skillsPrepares production launches. Use when preparing to deploy to production, or when asking what needs to be in place before shipping. Use when you need a pre-launch checklist, when setting up monitoring, when planning a staged rollout, or when you need a rollback strategy. | Stars: 119k | Updated | Claude CodeCodex | Verified | Install |
| childhoodthedotmack/claude-memBabysit a shipped release after merge, publication, or deployment. Verify published packages, production behavior, installed plugins, and late CI or user reports until the observation window is clean; use babysit for PR readiness before release. | Stars: 118k | Updated | Claude CodeCodex | Verified | Install |