Ar9av/PaperOrchestra / skills/content-refinement-agent
content-refinement-agent
VerifiedStep 5 of the PaperOrchestra pipeline (arXiv:2604.05018). Iteratively refine drafts/paper.tex by simulating peer review and applying targeted revisions, with strict accept/revert halt rules, deterministic 0-100 decision bands (Accept/Minor/Major/Reject) that drive a target-met early stop, and a Devil's Advocate concession-threshold guard that blocks acceptance on unresolved critical findings. Maintains a worklog and snapshots each iteration so revert is real, not symbolic. TRIGGER when the orchestrator delegates Step 5 or when the user asks to "refine the draft", "iterate on the paper", or "run peer review on this paper".
| File | Kind | Size |
|---|---|---|
| SKILL.md | skill md | 19.3 KB |
| references/ai-failure-modes.md | markdown | 813 B |
| references/claim-evidence-map.md | markdown | 3.1 KB |
| references/da-reviewer.md | markdown | 3.9 KB |
| references/halt-rules.md | markdown | 6.1 KB |
| references/prompt.md | markdown | 5.7 KB |
| references/reverse-outline.md | markdown | 3.0 KB |
| references/reviewer-rubric.md | markdown | 6.8 KB |
| references/safe-revision-rules.md | markdown | 4.8 KB |
| references/writing-quality-check.md | markdown | 720 B |
| scripts/apply_worklog.py | script | 3.1 KB |
| scripts/concession_guard.py | script | 7.5 KB |
| scripts/decision_band.py | script | 3.6 KB |
| scripts/reverse_outline.py | script | 10.3 KB |
| scripts/score_delta.py | script | 7.2 KB |
| scripts/score_trajectory.py | script | 8.6 KB |
| scripts/snapshot.py | script | 1.6 KB |
| scripts/test_reverse_outline.py | script | 5.7 KB |
| scripts/update_critique_memory.py | script | 11.1 KB |
references/halt-rules.md
View on GitHub# Halt Rules
Source: arXiv:2604.05018, §4 Step 5 ("Iterative Content Refinement"):
> After modifying the LaTeX source to address weaknesses, revisions are
> accepted if the overall score increases, or if it ties when net sub-axis
> gains are non-negative. The agent immediately reverts to the previous
> version and halts upon any overall score decrease, negative tie-breaker,
> or reaching the iteration limit.
Encoded as deterministic logic in `scripts/score_delta.py`. This file is the
human-readable specification.
## Definitions
Let:
- `prev` = score JSON from the previous accepted iteration
- `curr` = score JSON from the just-completed iteration
- `prev.overall` = `prev.overall_score`
- `curr.overall` = `curr.overall_score`
- `subaxis_delta(axis)` = `curr.axis_scores[axis].score - prev.axis_scores[axis].score`
- `net_subaxis_delta` = `sum(subaxis_delta(a) for a in 6 axes)`
## Decision rules (in order)
```
if curr.overall > prev.overall:
DECISION = ACCEPT_IMPROVED
elif curr.overall == prev.overall:
if net_subaxis_delta >= 0:
DECISION = ACCEPT_TIED_NON_NEGATIVE
else:
DECISION = REVERT_TIED_NEGATIVE_SUBAXIS
else: # curr.overall < prev.overall
DECISION = REVERT_OVERALL_DECREASED
```
The script exits with:
| Exit code | Meaning | Loop action |
|---|---|---|
| 0 | ACCEPT_IMPROVED | keep new draft, continue loop |
| 0 | ACCEPT_TIED_NON_NEGATIVE | keep new draft, continue loop |
| 1 | REVERT_OVERALL_DECREASED | rollback to prev, halt loop |
| 2 | REVERT_TIED_NEGATIVE_SUBAXIS | rollback to prev, halt loop |
| 4 | HALT_PLATEAU | keep new draft (accepted), halt loop |
| 5 | HALT_TARGET_MET | keep new draft (accepted), halt loop |
The script also prints a one-line decision string and a JSON object on
stdout for the host agent to log.
## Decision bands and the target-met halt
`score_delta.py` annotates every comparison with the prev/curr **decision band**
(`decision_band.py`): Accept (≥80), Minor Revision (65–79), Major Revision
(50–64), Reject (<50). These give the loop an *absolute* quality target on top of
the *relative* delta rules.
```
if DECISION in {ACCEPT_IMPROVED, ACCEPT_TIED_NON_NEGATIVE}:
if curr.overall >= accept_threshold (default 80):
DECISION = HALT_TARGET_MET # exit 5 — keep current draft, stop
elif consecutive_small >= plateau_streak:
DECISION = HALT_PLATEAU # exit 4 — keep current draft, stop
```
Target-met takes precedence over plateau: once the paper reaches the Accept
band there is no reason to keep iterating and risk a regression. Both 4 and 5
**keep** the just-accepted draft (unlike REVERT, which rolls back). Disable the
target-met halt with `--no-target-halt` (the band is still reported).
## Loop-level halt conditions
In addition to the per-iteration accept/revert decision, the loop halts
when ANY of these is true:
1. **Iteration cap reached.** Default 3 (configurable via env var
`PO_REFINE_MAX_ITER`). Per the paper Table 7, the typical
refinement count is "3× content refinement loop".
2. **REVERT decision** from `score_delta.py` (exit code 1 or 2).
3. **Empty weaknesses list.** If the simulated reviewer's `weaknesses`
array is empty, there is nothing to fix — halt.
4. **Plateau early-stop (exit code 4).** `score_delta.py` returns
`HALT_PLATEAU` when `N` consecutive accepted iterations each have
`overall_delta < threshold`. Default: threshold=1.0 points, N=3.
Configurable via `--plateau-threshold` and `--plateau-streak`.
5. **Target met (exit code 5).** `score_delta.py` returns `HALT_TARGET_MET`
when an accepted iteration reaches the Accept band (overall ≥ 80). The
current draft is promoted; the loop stops rather than risk a regression.
6. **DA CRITICAL standing (concession guard).** `concession_guard.py` exit 1
means a Devil's Advocate CRITICAL finding is still standing (unresolved and
not validly conceded). This **overrides an ACCEPT into a REVERT**: roll back
and require the next revision to address the finding before continuing. Exit
2 (a rejected concession with no blocked critical) is a WARN — the DA must
restate the attack, but it does not by itself force a revert. See
`da-reviewer.md`.
The calling loop must pass `--consecutive-small <count>` to
`score_delta.py` to track the streak across iterations:
```bash
CONSECUTIVE_SMALL=0
for iter in 1 2 3 ...; do
# ... run refinement LLM call ...
python score_delta.py \
--prev iter$((iter-1))/score.json \
--curr iter${iter}/score.json \
--plateau-threshold 1.0 \
--plateau-streak 3 \
--consecutive-small $CONSECUTIVE_SMALL
EXIT=$?
# Update streak counter from script output
CONSECUTIVE_SMALL=$(python -c "import json,sys; \
d=json.loads(open('iter${iter}/delta.json').read()); \
print(d['consecutive_small'])")
if [ $EXIT -ne 0 ]; then break; fi
done
```
**Why this matters**: in practice, ~85% of the refinement gain comes
in the first iteration (scores jump 5-8 points). Subsequent iterations
typically improve by <1 point. Without early-stop, the loop runs 3 full
LLM calls even when iterations 2 and 3 contribute near-zero value.
## Promoting the best snapshot
After halt, identify the iteration with the highest `accepted` overall
score:
```python
accepted_iters = [it for it in worklog.iterations if it.decision.startswith("ACCEPT")]
best = max(accepted_iters, key=lambda it: it.score.overall_score)
```
If the loop halted on REVERT, `best` is the iteration immediately *before*
the reverted one. Copy its `paper.tex` and `paper.pdf` to
`workspace/final/`.
## Worked example
Suppose:
| iter | overall | depth | exec | flow | clarity | evidence | style | decision |
|---|---|---|---|---|---|---|---|---|
| 0 | 64.5 | 65 | 70 | 60 | 55 | 72 | 68 | (baseline) |
| 1 | 67.3 | 68 | 73 | 64 | 58 | 74 | 70 | ACCEPT_IMPROVED |
| 2 | 67.3 | 70 | 73 | 64 | 58 | 73 | 71 | ACCEPT_TIED_NON_NEGATIVE (Σdelta = +2) |
| 3 | 66.0 | 70 | 70 | 62 | 56 | 73 | 71 | REVERT_OVERALL_DECREASED, HALT |
Promoted: iter 2 (`final/paper.tex` ← `iter2/paper.tex`).
Score trajectory in the run report:
```
64.5 → 67.3 (accept) → 67.3 (accept tied) → 66.0 (revert, halt)
```
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| writing-skillsobra/superpowersUse when creating new skills, editing existing skills, or verifying skills work before deployment | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |
| xlsxanthropics/skillsUse this skill any time a spreadsheet file is the primary input or output. This means any task where the user wants to: open, read, edit, or fix an existing .xlsx, .xlsm, .xltx, .csv, or .tsv file (e.g., adding columns, computing formulas, formatting, charting, cleaning messy data); create a new spreadsheet from scratch or from other data sources; or convert between tabular file formats. Trigger especially when the user references a spreadsheet file by name or path — even casually (like "the xlsx in my downloads") — and wants something done to it or produced from it. Also trigger for cleaning or restructuring messy tabular data files (malformed rows, misplaced headers, junk data) into proper spreadsheets. The deliverable must be a spreadsheet file. Do NOT trigger when the primary deliverable is a Word document, HTML report, standalone Python script, database pipeline, or Google Sheets API integration, even if tabular data is involved. | Stars: 198k | Updated | Claude CodeCodex | Verified | Install |
| shipping-and-launchaddyosmani/agent-skillsPrepares production launches. Use when preparing to deploy to production, or when asking what needs to be in place before shipping. Use when you need a pre-launch checklist, when setting up monitoring, when planning a staged rollout, or when you need a rollback strategy. | Stars: 119k | Updated | Claude CodeCodex | Verified | Install |
| childhoodthedotmack/claude-memBabysit a shipped release after merge, publication, or deployment. Verify published packages, production behavior, installed plugins, and late CI or user reports until the observation window is clean; use babysit for PR readiness before release. | Stars: 118k | Updated | Claude CodeCodex | Verified | Install |