Ar9av/PaperOrchestra / skills/literature-review-agent
literature-review-agent
UnverifiedStep 3 of the PaperOrchestra pipeline (arXiv:2604.05018). Execute the literature search strategy from outline.json — discover candidate papers via web search, verify them through Semantic Scholar (Levenshtein > 70 fuzzy title match, temporal cutoff, dedup by paperId), cross-corroborate against Crossref + OpenAlex to flag hallucinated citations, build a BibTeX file, and draft Introduction + Related Work using ≥90% of the verified pool. Runs in parallel with the plotting-agent. TRIGGER when the orchestrator delegates Step 3 or when the user asks to "find citations for my paper", "draft the related work", or "build the bibliography".
scripts/cross_verify.py
View on GitHub#!/usr/bin/env python3
"""
cross_verify.py — Cross-index corroboration for a verified citation pool.
Semantic Scholar verification (Levenshtein title match + cutoff + dedup) is the
pipeline's first gate. This script adds a second: every paper that S2 accepted
is re-checked against two *independent* scholarly indices — Crossref and
OpenAlex. A genuine paper turns up in all three with matching metadata; a
hallucinated or mis-attributed record typically does not. This is the practical
defense against the well-documented problem of fabricated citations leaking
into AI-assisted writing.
The script is a WARN gate, not a hard gate (mirrors validate_consistency.py):
it annotates the pool and writes a report, but exits non-zero only to draw the
host agent's attention to flagged entries — it does not block the pipeline.
The host agent reviews flagged citations and decides whether to drop them.
Confidence tiers written onto each paper's `cross_verification` field:
high corroborated by >=1 external index, no metadata conflicts
medium corroborated, but publication year disagrees beyond tolerance
low NOT found in Crossref or OpenAlex — phantom/hallucination risk
conflict a DOI present in the pool disagrees with the external index's DOI
Network etiquette: queries are throttled (--sleep, default 1.0s between papers)
and identify via a polite-pool email if PAPER_ORCHESTRA_MAILTO (or the
per-service CROSSREF_MAILTO / OPENALEX_MAILTO) is set. If an index is
unreachable the script degrades gracefully — it disables that index, notes it
in the report, and continues with whatever indices remain.
Usage:
python cross_verify.py --pool workspace/citation_pool.json
python cross_verify.py --pool workspace/citation_pool.json --inplace
python cross_verify.py --pool workspace/citation_pool.json \\
--out workspace/cross_verification_report.json \\
--indices crossref,openalex --threshold 70 --year-tolerance 1 --sleep 1.0
Exit codes:
0 every paper corroborated (all high/medium), no flags
1 one or more papers flagged low/conflict, OR an index was unavailable (WARN)
2 usage error / pool unreadable
"""
import argparse
import json
import os
import re
import sys
import time
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
import crossref_client # noqa: E402
import openalex_client # noqa: E402
try:
import Levenshtein # noqa: E402
def _ratio(a: str, b: str) -> int:
return int(round(Levenshtein.ratio(a, b) * 100))
except ImportError: # graceful fallback — stdlib only
from difflib import SequenceMatcher
def _ratio(a: str, b: str) -> int:
return int(round(SequenceMatcher(None, a, b).ratio() * 100))
def _normalize(s: str) -> str:
s = (s or "").lower().strip()
s = re.sub(r"[^a-z0-9\s]", " ", s)
return re.sub(r"\s+", " ", s).strip()
def title_ratio(a: str, b: str) -> int:
return _ratio(_normalize(a), _normalize(b))
def _bare_doi(doi: str | None) -> str:
if not doi:
return ""
d = doi.strip().lower()
for prefix in ("https://doi.org/", "http://doi.org/", "doi:"):
if d.startswith(prefix):
d = d[len(prefix):]
return d
def pool_doi(paper: dict) -> str:
"""Extract a DOI from a pool record, tolerating several shapes."""
ext = paper.get("externalIds") or {}
for k in ("DOI", "doi", "Doi"):
if ext.get(k):
return _bare_doi(ext[k])
if paper.get("doi"):
return _bare_doi(paper["doi"])
return ""
def best_match(title: str, hits: list[dict]) -> dict | None:
"""Pick the external hit whose title best matches `title`."""
best, best_r = None, -1
for h in hits:
r = title_ratio(title, h.get("title", ""))
if r > best_r:
best, best_r = h, r
if best is None:
return None
out = dict(best)
out["title_ratio"] = best_r
return out
def _safe(fn, *args):
"""Call a client function, converting its SystemExit (network/HTTP error)
into a None so cross_verify can degrade instead of dying."""
try:
return fn(*args)
except SystemExit:
return None
def check_index(module, paper: dict, threshold: int, strong_threshold: int, limit: int) -> dict:
"""Look one paper up in a single external index. Returns a per-index dict,
or {"available": False} if the index could not be reached.
Two match strengths are recorded:
found — title_ratio > threshold (lenient): "something like this exists"
strong — title_ratio >= strong_threshold, or an exact DOI hit: "this is
confidently the same paper". Only `strong` matches are trusted
for year/DOI conflict downgrades, so a noisy near-title hit
(e.g. a different paper with a similar name) cannot pollute the
metadata-conflict checks.
"""
title = paper.get("title", "")
doi = pool_doi(paper)
result = None
via = "title"
if doi:
r = _safe(module.lookup_doi, doi)
if r is None:
return {"available": False}
if r["data"]:
result = r["data"][0]
result["title_ratio"] = title_ratio(title, result.get("title", ""))
via = "doi"
if result is None:
r = _safe(module.search, title, limit)
if r is None:
return {"available": False}
result = best_match(title, r["data"])
if result is None:
return {"available": True, "found": False, "strong": False}
tr = result.get("title_ratio", 0)
found = tr > threshold or via == "doi"
strong = tr >= strong_threshold or via == "doi"
ext_doi = _bare_doi(result.get("doi"))
doi_state = "n/a"
if doi and ext_doi:
doi_state = "agree" if doi == ext_doi else "conflict"
return {
"available": True,
"found": bool(found),
"strong": bool(strong),
"via": via,
"matched_title": result.get("title", ""),
"title_ratio": tr,
"matched_year": result.get("year"),
"matched_doi": ext_doi,
"doi_state": doi_state,
}
def classify(paper: dict, per_index: dict, year_tolerance: int) -> dict:
found_in = [name for name, r in per_index.items() if r.get("found")]
# Only confident same-paper matches can trigger a metadata-conflict downgrade.
doi_conflict = any(
r.get("doi_state") == "conflict" and r.get("strong") for r in per_index.values()
)
year = paper.get("year")
year_conflict = False
if year is not None:
for r in per_index.values():
my = r.get("matched_year")
if r.get("strong") and my is not None and abs(int(my) - int(year)) > year_tolerance:
year_conflict = True
notes = []
if doi_conflict:
confidence = "conflict"
notes.append("DOI in pool disagrees with external index — verify this record")
elif not found_in:
confidence = "low"
notes.append("not found in Crossref or OpenAlex — possible phantom citation")
elif year_conflict:
confidence = "medium"
notes.append("corroborated, but publication year disagrees beyond tolerance")
else:
confidence = "high"
return {
"confidence": confidence,
"indices_found": found_in,
"doi_conflict": doi_conflict,
"year_conflict": year_conflict,
"per_index": per_index,
"notes": notes,
}
def main() -> int:
p = argparse.ArgumentParser(
description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter
)
p.add_argument("--pool", required=True, help="citation_pool.json path")
p.add_argument("--out", default=None,
help="Report path (default: <pool dir>/cross_verification_report.json)")
p.add_argument("--inplace", action="store_true",
help="Annotate each paper in the pool with its cross_verification result")
p.add_argument("--indices", default="crossref,openalex",
help="Comma-separated external indices to check (default both)")
p.add_argument("--threshold", type=int, default=70,
help="Levenshtein title-match threshold for corroboration "
"(default 70, matches S2 gate)")
p.add_argument("--strong-threshold", type=int, default=90,
help="Stricter title-match threshold before an external record's "
"year/DOI is trusted for a conflict downgrade (default 90)")
p.add_argument("--year-tolerance", type=int, default=1,
help="Allowed |year| difference before flagging (default 1)")
p.add_argument("--limit", type=int, default=5, help="Max hits per title search")
p.add_argument("--sleep", type=float, default=1.0,
help="Seconds to pause between papers (politeness, default 1.0)")
args = p.parse_args()
try:
with open(args.pool) as f:
pool = json.load(f)
except (OSError, json.JSONDecodeError) as exc:
print(f"ERROR: cannot read pool: {exc}", file=sys.stderr)
return 2
papers = pool.get("papers", [])
if not papers:
print("ERROR: pool['papers'] is empty or missing", file=sys.stderr)
return 2
modules = {"crossref": crossref_client, "openalex": openalex_client}
requested = [x.strip() for x in args.indices.split(",") if x.strip()]
unknown = [x for x in requested if x not in modules]
if unknown:
print(f"ERROR: unknown index/indices: {', '.join(unknown)}", file=sys.stderr)
return 2
disabled: set[str] = set()
summary = {"high": 0, "medium": 0, "low": 0, "conflict": 0}
flagged = []
for i, paper in enumerate(papers):
per_index = {}
for name in requested:
if name in disabled:
continue
res = check_index(modules[name], paper, args.threshold,
args.strong_threshold, args.limit)
if res.get("available") is False:
disabled.add(name)
print(f"WARN: index '{name}' unreachable — disabling for the rest of this run.",
file=sys.stderr)
continue
per_index[name] = res
verdict = classify(paper, per_index, args.year_tolerance)
summary[verdict["confidence"]] += 1
if verdict["confidence"] in ("low", "conflict"):
flagged.append({
"bibtex_key": paper.get("bibtex_key") or paper.get("key"),
"title": paper.get("title"),
"confidence": verdict["confidence"],
"notes": verdict["notes"],
})
if args.inplace:
paper["cross_verification"] = verdict
# Throttle between papers (skip after the last one).
if i < len(papers) - 1 and args.sleep > 0 and requested != list(disabled):
time.sleep(args.sleep)
report = {
"total_papers": len(papers),
"indices_checked": [x for x in requested if x not in disabled],
"indices_unavailable": sorted(disabled),
"summary": summary,
"flagged": flagged,
}
out_path = args.out or os.path.join(os.path.dirname(os.path.abspath(args.pool)),
"cross_verification_report.json")
with open(out_path, "w") as f:
json.dump(report, f, indent=2, ensure_ascii=False)
if args.inplace:
with open(args.pool, "w") as f:
json.dump(pool, f, indent=2, ensure_ascii=False)
# Human-readable summary to stdout.
print(f"Cross-index verification ({', '.join(report['indices_checked']) or 'none'}):")
print(f" {len(papers)} papers | high={summary['high']} "
f"medium={summary['medium']} low={summary['low']} conflict={summary['conflict']}")
print(f" report → {out_path}")
if flagged:
print(f"\nWARN: {len(flagged)} citation(s) need review:")
for f_ in flagged:
print(f" [{f_['confidence'].upper()}] {f_['bibtex_key']}: {f_['title']}")
for note in f_["notes"]:
print(f" → {note}")
print("\nReview these before building refs.bib. Drop any you cannot corroborate.")
if disabled:
print(f"\nNote: {', '.join(sorted(disabled))} unavailable this run — "
"corroboration is partial.", file=sys.stderr)
return 1 if (flagged or disabled) else 0
if __name__ == "__main__":
sys.exit(main())
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| writing-skillsobra/superpowersUse when creating new skills, editing existing skills, or verifying skills work before deployment | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |
| xlsxanthropics/skillsUse this skill any time a spreadsheet file is the primary input or output. This means any task where the user wants to: open, read, edit, or fix an existing .xlsx, .xlsm, .xltx, .csv, or .tsv file (e.g., adding columns, computing formulas, formatting, charting, cleaning messy data); create a new spreadsheet from scratch or from other data sources; or convert between tabular file formats. Trigger especially when the user references a spreadsheet file by name or path — even casually (like "the xlsx in my downloads") — and wants something done to it or produced from it. Also trigger for cleaning or restructuring messy tabular data files (malformed rows, misplaced headers, junk data) into proper spreadsheets. The deliverable must be a spreadsheet file. Do NOT trigger when the primary deliverable is a Word document, HTML report, standalone Python script, database pipeline, or Google Sheets API integration, even if tabular data is involved. | Stars: 198k | Updated | Claude CodeCodex | Verified | Install |
| shipping-and-launchaddyosmani/agent-skillsPrepares production launches. Use when preparing to deploy to production, or when asking what needs to be in place before shipping. Use when you need a pre-launch checklist, when setting up monitoring, when planning a staged rollout, or when you need a rollback strategy. | Stars: 119k | Updated | Claude CodeCodex | Verified | Install |
| childhoodthedotmack/claude-memBabysit a shipped release after merge, publication, or deployment. Verify published packages, production behavior, installed plugins, and late CI or user reports until the observation window is clean; use babysit for PR readiness before release. | Stars: 118k | Updated | Claude CodeCodex | Verified | Install |