Ar9av/PaperOrchestra / skills/content-refinement-agent
content-refinement-agent
VerifiedStep 5 of the PaperOrchestra pipeline (arXiv:2604.05018). Iteratively refine drafts/paper.tex by simulating peer review and applying targeted revisions, with strict accept/revert halt rules, deterministic 0-100 decision bands (Accept/Minor/Major/Reject) that drive a target-met early stop, and a Devil's Advocate concession-threshold guard that blocks acceptance on unresolved critical findings. Maintains a worklog and snapshots each iteration so revert is real, not symbolic. TRIGGER when the orchestrator delegates Step 5 or when the user asks to "refine the draft", "iterate on the paper", or "run peer review on this paper".
| File | Kind | Size |
|---|---|---|
| SKILL.md | skill md | 19.3 KB |
| references/ai-failure-modes.md | markdown | 813 B |
| references/claim-evidence-map.md | markdown | 3.1 KB |
| references/da-reviewer.md | markdown | 3.9 KB |
| references/halt-rules.md | markdown | 6.1 KB |
| references/prompt.md | markdown | 5.7 KB |
| references/reverse-outline.md | markdown | 3.0 KB |
| references/reviewer-rubric.md | markdown | 6.8 KB |
| references/safe-revision-rules.md | markdown | 4.8 KB |
| references/writing-quality-check.md | markdown | 720 B |
| scripts/apply_worklog.py | script | 3.1 KB |
| scripts/concession_guard.py | script | 7.5 KB |
| scripts/decision_band.py | script | 3.6 KB |
| scripts/reverse_outline.py | script | 10.3 KB |
| scripts/score_delta.py | script | 7.2 KB |
| scripts/score_trajectory.py | script | 8.6 KB |
| scripts/snapshot.py | script | 1.6 KB |
| scripts/test_reverse_outline.py | script | 5.7 KB |
| scripts/update_critique_memory.py | script | 11.1 KB |
scripts/concession_guard.py
View on GitHub#!/usr/bin/env python3
"""
concession_guard.py — Enforce the Devil's Advocate concession-threshold protocol
deterministically (references/da-reviewer.md).
The DA reviewer challenges the paper's core claims. A simulated reviewer left to
its own devices tends to *cave*: it concedes an attack as soon as the author
pushes back, even when the rebuttal is weak — sycophancy that defeats the point
of an adversarial reviewer. This script makes the protocol non-negotiable by
checking the DA's concession log against two hard rules, so the LLM cannot quietly
relax them:
Rule 1 — Concession requires evidence.
A finding may be conceded only if its rebuttal_score is >= 4 (the protocol's
"rebuttal directly / strongly addresses the attack with evidence"). Conceding
at rebuttal_score <= 3 is caving: the concession is REJECTED and the finding
is restored to "standing".
Rule 2 — No consecutive concessions (IRON RULE).
The DA may make at most one valid concession per two review rounds.
A concession in a round immediately following another conceding round is
REJECTED and the finding is restored to "standing".
A CRITICAL finding that is still standing after these rules (not resolved by the
revision, and not validly conceded) BLOCKS the "refinement accepted" decision —
the host must treat the iteration as REVERT, regardless of rubric scores.
Concession log schema (--log):
{
"rounds": [
{
"round": 1,
"findings": [
{
"id": "F1",
"severity": "critical" | "major" | "minor",
"attack": "Causal overclaiming: Sec 4 says X *causes* Y from corr only.",
"rebuttal_score": 2, # 1-5, DA's score of the author's rebuttal
"conceded": false, # did the DA drop the attack this round?
"resolved": false # was the underlying issue fixed in the revision?
}
]
}
]
}
Usage:
python concession_guard.py --log workspace/refinement/da_concessions.json
python concession_guard.py --log da_concessions.json --out guard_report.json
Exit codes:
0 CLEAR — no standing critical, no protocol violations → accept may proceed
1 BLOCK — a critical finding is still standing → host must REVERT this iteration
2 WARN — protocol violation(s) found but no critical blocked → DA must restate;
the rejected concession does not by itself force a revert
3 input / schema error
"""
import argparse
import json
import sys
VALID_SEVERITY = {"critical", "major", "minor"}
CONCESSION_MIN_REBUTTAL = 4 # rebuttal_score >= this to allow a concession
def analyze(rounds: list) -> dict:
violations = []
valid_concessions = []
standing_criticals = []
last_conceded_round = None # round index of the previous *valid* concession
for r in rounds:
rnum = r.get("round")
conceded_this_round = False
for fnd in r.get("findings", []):
fid = fnd.get("id", "?")
sev = fnd.get("severity", "minor")
conceded = bool(fnd.get("conceded", False))
resolved = bool(fnd.get("resolved", False))
score = fnd.get("rebuttal_score")
concession_valid = False
if conceded:
# Rule 1 — evidence threshold
if score is None or score < CONCESSION_MIN_REBUTTAL:
violations.append({
"round": rnum, "id": fid, "type": "caving",
"detail": f"conceded at rebuttal_score={score} "
f"(< {CONCESSION_MIN_REBUTTAL}); concession rejected",
})
# Rule 2 — no consecutive concessions
elif last_conceded_round is not None and rnum == last_conceded_round + 1:
violations.append({
"round": rnum, "id": fid, "type": "consecutive_concession",
"detail": f"concession in round {rnum} immediately follows a "
f"concession in round {last_conceded_round}; "
f"rejected (max one per two rounds)",
})
else:
concession_valid = True
conceded_this_round = True
valid_concessions.append({"round": rnum, "id": fid, "severity": sev})
# A critical is "standing" unless resolved OR validly conceded.
if sev == "critical" and not resolved and not concession_valid:
standing_criticals.append({
"round": rnum, "id": fid, "attack": fnd.get("attack", ""),
})
if conceded_this_round:
last_conceded_round = rnum
block = bool(standing_criticals)
if block:
action = "REVERT"
elif violations:
action = "DA_RESTATE"
else:
action = "PROCEED"
return {
"rounds_analyzed": len(rounds),
"valid_concessions": valid_concessions,
"violations": violations,
"standing_criticals": standing_criticals,
"block_accept": block,
"recommended_action": action,
}
def main() -> int:
p = argparse.ArgumentParser(description=__doc__,
formatter_class=argparse.RawDescriptionHelpFormatter)
p.add_argument("--log", required=True, help="Concession log JSON path")
p.add_argument("--out", default=None, help="Optional path to write the guard report JSON")
args = p.parse_args()
try:
with open(args.log) as f:
log = json.load(f)
except (OSError, json.JSONDecodeError) as e:
print(f"ERROR: cannot read concession log: {e}", file=sys.stderr)
return 3
rounds = log.get("rounds")
if not isinstance(rounds, list) or not rounds:
print("ERROR: log['rounds'] must be a non-empty list", file=sys.stderr)
return 3
# Light schema validation — fail loudly rather than silently mis-classify.
for r in rounds:
for fnd in r.get("findings", []):
sev = fnd.get("severity", "minor")
if sev not in VALID_SEVERITY:
print(f"ERROR: finding {fnd.get('id','?')} has invalid severity "
f"'{sev}' (expected one of {sorted(VALID_SEVERITY)})", file=sys.stderr)
return 3
score = fnd.get("rebuttal_score")
if score is not None and not (1 <= score <= 5):
print(f"ERROR: finding {fnd.get('id','?')} rebuttal_score={score} "
f"out of range 1-5", file=sys.stderr)
return 3
report = analyze(rounds)
if args.out:
with open(args.out, "w") as f:
json.dump(report, f, indent=2, ensure_ascii=False)
# Human-readable summary.
print(f"DA concession guard: {report['rounds_analyzed']} round(s) "
f"valid_concessions={len(report['valid_concessions'])} "
f"violations={len(report['violations'])} "
f"standing_criticals={len(report['standing_criticals'])}")
for v in report["violations"]:
print(f" VIOLATION [{v['type']}] round {v['round']} {v['id']}: {v['detail']}")
for c in report["standing_criticals"]:
print(f" STANDING CRITICAL round {c['round']} {c['id']}: {c['attack']}")
print(f" → recommended action: {report['recommended_action']}")
if report["block_accept"]:
return 1
if report["violations"]:
return 2
return 0
if __name__ == "__main__":
sys.exit(main())
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| writing-skillsobra/superpowersUse when creating new skills, editing existing skills, or verifying skills work before deployment | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |
| xlsxanthropics/skillsUse this skill any time a spreadsheet file is the primary input or output. This means any task where the user wants to: open, read, edit, or fix an existing .xlsx, .xlsm, .xltx, .csv, or .tsv file (e.g., adding columns, computing formulas, formatting, charting, cleaning messy data); create a new spreadsheet from scratch or from other data sources; or convert between tabular file formats. Trigger especially when the user references a spreadsheet file by name or path — even casually (like "the xlsx in my downloads") — and wants something done to it or produced from it. Also trigger for cleaning or restructuring messy tabular data files (malformed rows, misplaced headers, junk data) into proper spreadsheets. The deliverable must be a spreadsheet file. Do NOT trigger when the primary deliverable is a Word document, HTML report, standalone Python script, database pipeline, or Google Sheets API integration, even if tabular data is involved. | Stars: 197k | Updated | Claude CodeCodex | Verified | Install |
| shipping-and-launchaddyosmani/agent-skillsPrepares production launches. Use when preparing to deploy to production, or when asking what needs to be in place before shipping. Use when you need a pre-launch checklist, when setting up monitoring, when planning a staged rollout, or when you need a rollback strategy. | Stars: 119k | Updated | Claude CodeCodex | Verified | Install |
| childhoodthedotmack/claude-memBabysit a shipped release after merge, publication, or deployment. Verify published packages, production behavior, installed plugins, and late CI or user reports until the observation window is clean; use babysit for PR readiness before release. | Stars: 118k | Updated | Claude CodeCodex | Verified | Install |