Ar9av/PaperOrchestra / skills/content-refinement-agent
content-refinement-agent
VerifiedStep 5 of the PaperOrchestra pipeline (arXiv:2604.05018). Iteratively refine drafts/paper.tex by simulating peer review and applying targeted revisions, with strict accept/revert halt rules, deterministic 0-100 decision bands (Accept/Minor/Major/Reject) that drive a target-met early stop, and a Devil's Advocate concession-threshold guard that blocks acceptance on unresolved critical findings. Maintains a worklog and snapshots each iteration so revert is real, not symbolic. TRIGGER when the orchestrator delegates Step 5 or when the user asks to "refine the draft", "iterate on the paper", or "run peer review on this paper".
| File | Kind | Size |
|---|---|---|
| SKILL.md | skill md | 19.3 KB |
| references/ai-failure-modes.md | markdown | 813 B |
| references/claim-evidence-map.md | markdown | 3.1 KB |
| references/da-reviewer.md | markdown | 3.9 KB |
| references/halt-rules.md | markdown | 6.1 KB |
| references/prompt.md | markdown | 5.7 KB |
| references/reverse-outline.md | markdown | 3.0 KB |
| references/reviewer-rubric.md | markdown | 6.8 KB |
| references/safe-revision-rules.md | markdown | 4.8 KB |
| references/writing-quality-check.md | markdown | 720 B |
| scripts/apply_worklog.py | script | 3.1 KB |
| scripts/concession_guard.py | script | 7.5 KB |
| scripts/decision_band.py | script | 3.6 KB |
| scripts/reverse_outline.py | script | 10.3 KB |
| scripts/score_delta.py | script | 7.2 KB |
| scripts/score_trajectory.py | script | 8.6 KB |
| scripts/snapshot.py | script | 1.6 KB |
| scripts/test_reverse_outline.py | script | 5.7 KB |
| scripts/update_critique_memory.py | script | 11.1 KB |
references/claim-evidence-map.md
View on GitHub# Claim-Evidence Map
Every quantitative statement in a paper is one of three things:
1. **Our result** — the number must appear in `experimental_log.md`.
2. **Someone else's result** — the sentence must carry a citation.
3. **A fabrication** — the number appears in neither place.
The third category is the failure mode that ends careers, and it is the one a
language model produces most naturally: a plausible-looking figure generated
to make a sentence land. Treating claim-evidence alignment as a hard
constraint — rather than as something the reviewer might notice — is the
single highest-value gate in the refinement loop.
## Running it
```bash
python skills/paper-orchestra/scripts/claim_evidence_gate.py \
--paper workspace/drafts/paper.tex \
--log workspace/inputs/experimental_log.md \
--out workspace/claim_evidence_report.json \
--out-md workspace/claim_evidence_map.md
```
Exit codes: `0` PASS, `1` WARN (unsupported claims present), `2` ERROR (input
missing). This is a WARN gate — it never halts the pipeline.
## The three statuses
| Status | Meaning | Action |
|---|---|---|
| `supported` | The value occurs in `experimental_log.md` | None |
| `attributed` | The sentence carries a `\cite`, or a prior-work cue such as "prior work" / "et al." / "according to" | Spot-check that the citation actually contains the number |
| `needs evidence` | Neither | Verify against the log, attribute it to a source, or remove it |
`needs evidence` does not mean "wrong". A number computed from two logged
values (a ratio, a percentage delta) is legitimately absent from the log. The
gate cannot arithmetic, so it reports; the revision agent decides. What the
gate guarantees is that no such number passes through unexamined.
## Using the map in the loop
Read `workspace/claim_evidence_map.md` at Step 0 and pass every
`needs evidence` row to the revision agent as an explicit agenda item:
> The following values appear in the paper but cannot be corroborated in
> `experimental_log.md` and carry no citation. For each: restate it from
> logged values, attribute it to a cited source, or remove the claim.
> Do not weaken the sentence into vagueness to make the number defensible.
Re-run the gate after the iteration. A `needs evidence` row that survives two
iterations is a signal to delete the sentence, not to keep rewording it.
## What the gate does not check
- Whether a number is **correctly derived** from the logged values.
- Whether the cited paper actually reports the attributed number.
- Whether a **qualitative** claim ("substantially outperforms") is supported.
The five-dimension self-review in `references/reviewer-rubric.md` covers
those; this gate only covers numbers.
## Provenance
The claim-evidence contract (`Claim | Evidence | Status`, with
claim-evidence alignment treated as a hard constraint for Abstract and
Introduction) follows the output contract in
[Master-cai/Research-Paper-Writing-Skills](https://github.com/Master-cai/Research-Paper-Writing-Skills)
(MIT). The deterministic extraction, LaTeX normalization, and attribution
detection are PaperOrchestra's.
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| writing-skillsobra/superpowersUse when creating new skills, editing existing skills, or verifying skills work before deployment | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |
| xlsxanthropics/skillsUse this skill any time a spreadsheet file is the primary input or output. This means any task where the user wants to: open, read, edit, or fix an existing .xlsx, .xlsm, .xltx, .csv, or .tsv file (e.g., adding columns, computing formulas, formatting, charting, cleaning messy data); create a new spreadsheet from scratch or from other data sources; or convert between tabular file formats. Trigger especially when the user references a spreadsheet file by name or path — even casually (like "the xlsx in my downloads") — and wants something done to it or produced from it. Also trigger for cleaning or restructuring messy tabular data files (malformed rows, misplaced headers, junk data) into proper spreadsheets. The deliverable must be a spreadsheet file. Do NOT trigger when the primary deliverable is a Word document, HTML report, standalone Python script, database pipeline, or Google Sheets API integration, even if tabular data is involved. | Stars: 198k | Updated | Claude CodeCodex | Verified | Install |
| shipping-and-launchaddyosmani/agent-skillsPrepares production launches. Use when preparing to deploy to production, or when asking what needs to be in place before shipping. Use when you need a pre-launch checklist, when setting up monitoring, when planning a staged rollout, or when you need a rollback strategy. | Stars: 119k | Updated | Claude CodeCodex | Verified | Install |
| childhoodthedotmack/claude-memBabysit a shipped release after merge, publication, or deployment. Verify published packages, production behavior, installed plugins, and late CI or user reports until the observation window is clean; use babysit for PR readiness before release. | Stars: 118k | Updated | Claude CodeCodex | Verified | Install |