Skip to content

Ar9av/PaperOrchestra / skills/content-refinement-agent

content-refinement-agent

Verified

Step 5 of the PaperOrchestra pipeline (arXiv:2604.05018). Iteratively refine drafts/paper.tex by simulating peer review and applying targeted revisions, with strict accept/revert halt rules, deterministic 0-100 decision bands (Accept/Minor/Major/Reject) that drive a target-met early stop, and a Devil's Advocate concession-threshold guard that blocks acceptance on unresolved critical findings. Maintains a worklog and snapshots each iteration so revert is real, not symbolic. TRIGGER when the orchestrator delegates Step 5 or when the user asks to "refine the draft", "iterate on the paper", or "run peer review on this paper".

scripts/score_delta.py

View on GitHub
#!/usr/bin/env python3
"""
score_delta.py — Apply the PaperOrchestra refinement halt rules from two
score JSONs.

Encodes the halt rules from arXiv:2604.05018 §4 Step 5:

  - ACCEPT if curr.overall > prev.overall
  - ACCEPT if curr.overall == prev.overall AND net sub-axis delta >= 0
  - REVERT (overall_decreased) if curr.overall < prev.overall
  - REVERT (tied_negative_subaxis) if curr.overall == prev.overall AND
            net sub-axis delta < 0

Additionally encodes the plateau early-stop rule (not in the original paper
but added to match its cost budget of ~5-7 LLM calls):

  - HALT_PLATEAU if the improvement is accepted but overall_delta is below
    --plateau-threshold for --plateau-streak or more consecutive iterations.
    Exit code 4.  The loop should stop — further iterations are unlikely to
    yield meaningful gains.

And the decision-band target halt (see decision_band.py):

  - HALT_TARGET_MET if an accepted iteration reaches the "Accept" band
    (overall >= --accept-threshold, default 80). Exit code 5. The paper now
    clears the acceptance bar, so the loop stops on the current draft rather
    than risk regressing it chasing marginal gains. Takes precedence over the
    plateau halt. Disable with --no-target-halt.

Every output JSON also carries the prev/curr decision band (Accept / Minor
Revision / Major Revision / Reject) so the run report can show an absolute
quality trajectory, not just relative deltas.

Exit codes:
    0  ACCEPT (improved or tied non-negative; below Accept band; no plateau)
    1  REVERT (overall decreased)
    2  REVERT (tied with negative sub-axis delta)
    3  argument or input error
    4  HALT_PLATEAU (accepted but diminishing returns detected)
    5  HALT_TARGET_MET (accepted and reached the Accept band)

Score JSON shape (see references/reviewer-rubric.md):
    {
      "axis_scores": {
        "scientific_depth":     {"score": 65, ...},
        "technical_execution":  {"score": 70, ...},
        "logical_flow":         {"score": 60, ...},
        "writing_clarity":      {"score": 55, ...},
        "evidence_presentation":{"score": 72, ...},
        "academic_style":       {"score": 68, ...}
      },
      "overall_score": 64.5,
      ...
    }

Usage:
    python score_delta.py --prev iter0/score.json --curr iter1/score.json
    python score_delta.py --prev iter2/score.json --curr iter3/score.json \\
        --plateau-threshold 1.0 --plateau-streak 2 --consecutive-small 2
"""
import argparse
import json
import os
import sys

sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
from decision_band import band_for, BAND_RANK  # noqa: E402

AXES = [
    "scientific_depth",
    "technical_execution",
    "logical_flow",
    "writing_clarity",
    "evidence_presentation",
    "academic_style",
]

DEFAULT_PLATEAU_THRESHOLD = 1.0  # points
DEFAULT_PLATEAU_STREAK = 3       # consecutive iterations below threshold → halt


def load(path: str) -> dict:
    with open(path) as f:
        return json.load(f)


def main() -> int:
    p = argparse.ArgumentParser(description=__doc__)
    p.add_argument("--prev", required=True, help="Score JSON from previous accepted iteration")
    p.add_argument("--curr", required=True, help="Score JSON from just-completed iteration")
    p.add_argument(
        "--plateau-threshold", type=float, default=DEFAULT_PLATEAU_THRESHOLD,
        metavar="POINTS",
        help=f"Minimum overall_delta to not count as a 'small' improvement "
             f"(default: {DEFAULT_PLATEAU_THRESHOLD})",
    )
    p.add_argument(
        "--plateau-streak", type=int, default=DEFAULT_PLATEAU_STREAK,
        metavar="N",
        help=f"Number of consecutive small improvements before HALT_PLATEAU "
             f"(default: {DEFAULT_PLATEAU_STREAK})",
    )
    p.add_argument(
        "--consecutive-small", type=int, default=0,
        metavar="N",
        help="Number of consecutive small-delta accepted iterations so far "
             "(maintained by the calling loop; default: 0)",
    )
    p.add_argument(
        "--accept-threshold", type=float, default=80.0, metavar="SCORE",
        help="Overall score at/above which the paper is in the Accept band; "
             "an accepted iteration here triggers HALT_TARGET_MET (default: 80)",
    )
    p.add_argument(
        "--no-target-halt", action="store_true",
        help="Do not halt on reaching the Accept band (still reports the band)",
    )
    args = p.parse_args()

    try:
        prev = load(args.prev)
        curr = load(args.curr)
    except (OSError, json.JSONDecodeError) as e:
        print(f"ERROR: failed to load score JSONs: {e}", file=sys.stderr)
        return 3

    p_overall = float(prev.get("overall_score", 0))
    c_overall = float(curr.get("overall_score", 0))
    overall_delta = c_overall - p_overall

    p_axes = prev.get("axis_scores") or {}
    c_axes = curr.get("axis_scores") or {}
    deltas: dict[str, float] = {}
    for ax in AXES:
        ps = float((p_axes.get(ax) or {}).get("score", 0))
        cs = float((c_axes.get(ax) or {}).get("score", 0))
        deltas[ax] = cs - ps
    net_subaxis = sum(deltas.values())

    # --- Primary accept/revert decision ---
    if c_overall > p_overall:
        decision = "ACCEPT_IMPROVED"
        exit_code = 0
    elif c_overall == p_overall:
        if net_subaxis >= 0:
            decision = "ACCEPT_TIED_NON_NEGATIVE"
            exit_code = 0
        else:
            decision = "REVERT_TIED_NEGATIVE_SUBAXIS"
            exit_code = 2
    else:
        decision = "REVERT_OVERALL_DECREASED"
        exit_code = 1

    # --- Decision bands (absolute quality, independent of the delta) ---
    band_prev = band_for(p_overall, args.accept_threshold)
    band_curr = band_for(c_overall, args.accept_threshold)

    # --- Plateau early-stop (only applies to accepted iterations) ---
    is_small_delta = overall_delta < args.plateau_threshold
    new_consecutive_small = (args.consecutive_small + 1) if is_small_delta else 0
    plateau_triggered = False

    # --- Target-met halt takes precedence over plateau ---
    target_met = False
    if exit_code == 0 and not args.no_target_halt and c_overall >= args.accept_threshold:
        decision = "HALT_TARGET_MET"
        exit_code = 5
        target_met = True
    elif exit_code == 0 and new_consecutive_small >= args.plateau_streak:
        decision = "HALT_PLATEAU"
        exit_code = 4
        plateau_triggered = True

    out = {
        "decision":            decision,
        "exit_code":           exit_code,
        "overall_prev":        p_overall,
        "overall_curr":        c_overall,
        "overall_delta":       overall_delta,
        "decision_band_prev":  band_prev,
        "decision_band_curr":  band_curr,
        "band_improved":       BAND_RANK[band_curr] > BAND_RANK[band_prev],
        "target_met":          target_met,
        "accept_threshold":    args.accept_threshold,
        "subaxis_deltas":      deltas,
        "net_subaxis":         net_subaxis,
        "is_small_delta":      is_small_delta,
        "consecutive_small":   new_consecutive_small,
        "plateau_threshold":   args.plateau_threshold,
        "plateau_streak":      args.plateau_streak,
        "plateau_triggered":   plateau_triggered,
    }
    print(json.dumps(out, indent=2))
    return exit_code


if __name__ == "__main__":
    sys.exit(main())
writing-skillsobra/superpowersUse when creating new skills, editing existing skills, or verifying skills work before deploymentStars: 322kUpdated Claude CodeCodexVerifiedInstall
xlsxanthropics/skillsUse this skill any time a spreadsheet file is the primary input or output. This means any task where the user wants to: open, read, edit, or fix an existing .xlsx, .xlsm, .xltx, .csv, or .tsv file (e.g., adding columns, computing formulas, formatting, charting, cleaning messy data); create a new spreadsheet from scratch or from other data sources; or convert between tabular file formats. Trigger especially when the user references a spreadsheet file by name or path — even casually (like "the xlsx in my downloads") — and wants something done to it or produced from it. Also trigger for cleaning or restructuring messy tabular data files (malformed rows, misplaced headers, junk data) into proper spreadsheets. The deliverable must be a spreadsheet file. Do NOT trigger when the primary deliverable is a Word document, HTML report, standalone Python script, database pipeline, or Google Sheets API integration, even if tabular data is involved.Stars: 197kUpdated Claude CodeCodexVerifiedInstall
shipping-and-launchaddyosmani/agent-skillsPrepares production launches. Use when preparing to deploy to production, or when asking what needs to be in place before shipping. Use when you need a pre-launch checklist, when setting up monitoring, when planning a staged rollout, or when you need a rollback strategy.Stars: 119kUpdated Claude CodeCodexVerifiedInstall
childhoodthedotmack/claude-memBabysit a shipped release after merge, publication, or deployment. Verify published packages, production behavior, installed plugins, and late CI or user reports until the observation window is clean; use babysit for PR readiness before release.Stars: 118kUpdated Claude CodeCodexVerifiedInstall