Skip to content

YCC-Lover/research-starter-skills / skills/research-starter-paper

research-starter-paper

Unverified

科研学习与论文写作总入口,协调六个模块。帮助不知道提供什么材料的新手逐步产出、自动保存和恢复项目进度,也用于跨阶段研究任务;区分学习、执行和只读检查。

FileKindSize
SKILL.mdskill md8.8 KB
agents/openai.yamlconfig366 B
assets/evidence-ledger.tsvother106 B
assets/manuscript-plan.mdmarkdown423 B
assets/project-state.mdmarkdown928 B
assets/requirements-research.txtmarkdown263 B
assets/task-brief.mdmarkdown429 B
references/advanced-capabilities.jsonconfig8.5 KB
references/advanced-research.mdmarkdown8.5 KB
references/discipline-fit.mdmarkdown2.3 KB
references/evidence-and-figures.mdmarkdown2.8 KB
references/guided-start.mdmarkdown7.2 KB
references/handoff.mdmarkdown1.9 KB
references/learning.mdmarkdown3.2 KB
references/paper-sections.mdmarkdown5.6 KB
references/research-engine-notice.mdmarkdown1.6 KB
references/research-engine-patches.jsonconfig3.4 KB
references/research-practice.mdmarkdown3.2 KB
references/research.mdmarkdown3.2 KB
references/revision.mdmarkdown2.1 KB
references/runtime-helpers.mdmarkdown6.0 KB
references/source-manifest.jsonconfig34.8 KB
references/sources.mdmarkdown7.4 KB
references/task-brief.mdmarkdown1.6 KB
resources/research-engine/LICENSEother19.1 KB
resources/research-engine/MODE_REGISTRY.mdmarkdown6.7 KB
resources/research-engine/NOTICE.mdmarkdown1.0 KB
resources/research-engine/THIRD_PARTY.mdmarkdown5.0 KB
resources/research-engine/academic-paper-reviewer/WORKFLOW.mdmarkdown43.9 KB
resources/research-engine/academic-paper-reviewer/agents/devils_advocate_reviewer_agent.mdmarkdown42.0 KB
resources/research-engine/academic-paper-reviewer/agents/domain_reviewer_agent.mdmarkdown32.4 KB
resources/research-engine/academic-paper-reviewer/agents/editorial_synthesizer_agent.mdmarkdown41.8 KB
resources/research-engine/academic-paper-reviewer/agents/eic_agent.mdmarkdown28.1 KB
resources/research-engine/academic-paper-reviewer/agents/field_analyst_agent.mdmarkdown11.2 KB
resources/research-engine/academic-paper-reviewer/agents/methodology_reviewer_agent.mdmarkdown44.0 KB
resources/research-engine/academic-paper-reviewer/agents/perspective_reviewer_agent.mdmarkdown32.7 KB
resources/research-engine/academic-paper-reviewer/examples/hei_paper_review_example.mdmarkdown33.8 KB
resources/research-engine/academic-paper-reviewer/examples/interdisciplinary_review_example.mdmarkdown27.6 KB
resources/research-engine/academic-paper-reviewer/examples/subclaim_decomposition_example.mdmarkdown8.3 KB
resources/research-engine/academic-paper-reviewer/references/calibration_mode_protocol.mdmarkdown15.5 KB
resources/research-engine/academic-paper-reviewer/references/changelog.mdmarkdown1.4 KB
resources/research-engine/academic-paper-reviewer/references/editorial_decision_standards.mdmarkdown12.8 KB
resources/research-engine/academic-paper-reviewer/references/guided_mode_protocol.mdmarkdown2.0 KB
resources/research-engine/academic-paper-reviewer/references/integration_guide.mdmarkdown680 B
resources/research-engine/academic-paper-reviewer/references/quality_rubrics.mdmarkdown6.3 KB
resources/research-engine/academic-paper-reviewer/references/re_review_mode_protocol.mdmarkdown51.9 KB
resources/research-engine/academic-paper-reviewer/references/review_criteria_framework.mdmarkdown6.2 KB
resources/research-engine/academic-paper-reviewer/references/review_panel_provenance_protocol.mdmarkdown9.4 KB
resources/research-engine/academic-paper-reviewer/references/review_quality_thinking.mdmarkdown3.3 KB
resources/research-engine/academic-paper-reviewer/references/reviewer_sprint_prompt_source.mdmarkdown47.9 KB
resources/research-engine/academic-paper-reviewer/references/sprint_contract_protocol.mdmarkdown32.1 KB
resources/research-engine/academic-paper-reviewer/references/statistical_reporting_standards.mdmarkdown29.8 KB
resources/research-engine/academic-paper-reviewer/references/top_journals_by_field.mdmarkdown12.6 KB
resources/research-engine/academic-paper-reviewer/templates/editorial_decision_template.mdmarkdown11.0 KB
resources/research-engine/academic-paper-reviewer/templates/peer_review_report_template.mdmarkdown14.8 KB
resources/research-engine/academic-paper-reviewer/templates/revision_response_template.mdmarkdown7.4 KB
resources/research-engine/academic-paper/WORKFLOW.mdmarkdown63.8 KB
resources/research-engine/academic-paper/agents/abstract_bilingual_agent.mdmarkdown12.0 KB
resources/research-engine/academic-paper/agents/argument_builder_agent.mdmarkdown10.9 KB
resources/research-engine/academic-paper/agents/citation_compliance_agent.mdmarkdown23.5 KB
resources/research-engine/academic-paper/agents/draft_writer_agent.mdmarkdown56.1 KB
resources/research-engine/academic-paper/agents/formatter_agent.mdmarkdown63.8 KB
resources/research-engine/academic-paper/agents/intake_agent.mdmarkdown25.5 KB
resources/research-engine/academic-paper/agents/literature_strategist_agent.mdmarkdown46.5 KB
resources/research-engine/academic-paper/agents/peer_reviewer_agent.mdmarkdown29.5 KB
resources/research-engine/academic-paper/agents/revision_coach_agent.mdmarkdown20.0 KB
resources/research-engine/academic-paper/agents/socratic_mentor_agent.mdmarkdown27.9 KB
resources/research-engine/academic-paper/agents/structure_architect_agent.mdmarkdown18.9 KB
resources/research-engine/academic-paper/agents/visualization_agent.mdmarkdown17.2 KB
resources/research-engine/academic-paper/examples/chinese_paper_example.mdmarkdown21.1 KB
resources/research-engine/academic-paper/examples/clinical_citation_verification_checklist.mdmarkdown4.4 KB
resources/research-engine/academic-paper/examples/clinical_epistemic_status_example.mdmarkdown6.6 KB
resources/research-engine/academic-paper/examples/commitment_ledger_example.mdmarkdown8.1 KB
resources/research-engine/academic-paper/examples/imrad_hei_example.mdmarkdown23.3 KB
resources/research-engine/academic-paper/examples/literature_review_example.mdmarkdown30.8 KB
resources/research-engine/academic-paper/examples/plan_mode_guided_writing.mdmarkdown34.3 KB
resources/research-engine/academic-paper/examples/revision_mode_example.mdmarkdown20.1 KB
resources/research-engine/academic-paper/examples/revision_recovery_example.mdmarkdown41.2 KB
resources/research-engine/academic-paper/examples/version_family_reconciliation_example.mdmarkdown3.8 KB
resources/research-engine/academic-paper/references/abstract_writing_guide.mdmarkdown12.5 KB
resources/research-engine/academic-paper/references/academic_writing_style.mdmarkdown7.6 KB
resources/research-engine/academic-paper/references/anti_leakage_protocol.mdmarkdown4.4 KB
resources/research-engine/academic-paper/references/apa7_chinese_citation_guide.mdmarkdown14.0 KB
resources/research-engine/academic-paper/references/apa7_extended_guide.mdmarkdown6.6 KB
resources/research-engine/academic-paper/references/changelog.mdmarkdown2.8 KB
resources/research-engine/academic-paper/references/citation_format_switcher.mdmarkdown7.4 KB
resources/research-engine/academic-paper/references/committee_correspondence_protocol.mdmarkdown6.3 KB
resources/research-engine/academic-paper/references/credit_authorship_guide.mdmarkdown13.7 KB
resources/research-engine/academic-paper/references/disclosure_mode_protocol.mdmarkdown53.9 KB
resources/research-engine/academic-paper/references/domain_evidence_profiles.mdmarkdown4.2 KB
resources/research-engine/academic-paper/references/failure_paths.mdmarkdown14.2 KB
resources/research-engine/academic-paper/references/funding_statement_guide.mdmarkdown11.9 KB
resources/research-engine/academic-paper/references/hei_domain_glossary.mdmarkdown6.8 KB
resources/research-engine/academic-paper/references/intro_title_rhetoric_guide.mdmarkdown8.7 KB
resources/research-engine/academic-paper/references/journal_submission_guide.mdmarkdown8.7 KB
resources/research-engine/academic-paper/references/latex_template_reference.mdmarkdown9.6 KB
resources/research-engine/academic-paper/references/mode_selection_guide.mdmarkdown17.5 KB
resources/research-engine/academic-paper/references/paper_structure_patterns.mdmarkdown8.4 KB
resources/research-engine/academic-paper/references/plan_mode_protocol.mdmarkdown6.7 KB
resources/research-engine/academic-paper/references/policy_anchor_disclosure_protocol.mdmarkdown21.0 KB
resources/research-engine/academic-paper/references/policy_anchor_table.mdmarkdown21.2 KB
resources/research-engine/academic-paper/references/revision_patch_protocol.mdmarkdown12.9 KB
resources/research-engine/academic-paper/references/statistical_visualization_standards.mdmarkdown23.7 KB
resources/research-engine/academic-paper/references/venue_disclosure_policies.mdmarkdown38.4 KB
resources/research-engine/academic-paper/references/vlm_figure_verification.mdmarkdown9.1 KB
resources/research-engine/academic-paper/references/workflow_phase_details.mdmarkdown4.7 KB
resources/research-engine/academic-paper/references/writing_judgment_framework.mdmarkdown3.3 KB
resources/research-engine/academic-paper/references/writing_quality_check.mdmarkdown13.0 KB
resources/research-engine/academic-paper/templates/bilingual_abstract_template.mdmarkdown4.7 KB
resources/research-engine/academic-paper/templates/case_study_template.mdmarkdown2.8 KB
resources/research-engine/academic-paper/templates/conference_paper_template.mdmarkdown2.3 KB
resources/research-engine/academic-paper/templates/credit_statement_template.mdmarkdown4.4 KB
resources/research-engine/academic-paper/templates/funding_statement_template.mdmarkdown8.9 KB
resources/research-engine/academic-paper/templates/imrad_template.mdmarkdown4.4 KB
resources/research-engine/academic-paper/templates/latex_article_template.texother4.7 KB
resources/research-engine/academic-paper/templates/literature_review_template.mdmarkdown2.8 KB
resources/research-engine/academic-paper/templates/policy_brief_template.mdmarkdown3.6 KB
resources/research-engine/academic-paper/templates/revision_tracking_template.mdmarkdown10.2 KB
resources/research-engine/academic-paper/templates/theoretical_paper_template.mdmarkdown2.8 KB
resources/research-engine/academic-pipeline/WORKFLOW.mdmarkdown58.0 KB
resources/research-engine/academic-pipeline/agents/claim_ref_alignment_audit_agent.mdmarkdown44.3 KB
resources/research-engine/academic-pipeline/agents/collaboration_depth_agent.mdmarkdown9.8 KB
resources/research-engine/academic-pipeline/agents/integrity_verification_agent.mdmarkdown70.7 KB
resources/research-engine/academic-pipeline/agents/pipeline_orchestrator_agent.mdmarkdown173.8 KB
resources/research-engine/academic-pipeline/agents/state_tracker_agent.mdmarkdown26.8 KB
resources/research-engine/academic-pipeline/examples/full_pipeline_example.mdmarkdown17.7 KB
resources/research-engine/academic-pipeline/examples/integrity_failure_recovery.mdmarkdown26.8 KB
resources/research-engine/academic-pipeline/examples/mid_entry_example.mdmarkdown13.4 KB
resources/research-engine/academic-pipeline/references/adapters/.gitkeepother0 B
resources/research-engine/academic-pipeline/references/adapters/overview.mdmarkdown10.0 KB
resources/research-engine/academic-pipeline/references/ai_research_failure_modes.mdmarkdown15.6 KB
resources/research-engine/academic-pipeline/references/changelog.mdmarkdown5.8 KB
resources/research-engine/academic-pipeline/references/claim_audit_calibration_protocol.mdmarkdown14.2 KB
resources/research-engine/academic-pipeline/references/claim_verification_protocol.mdmarkdown27.2 KB
resources/research-engine/academic-pipeline/references/external_review_protocol.mdmarkdown6.2 KB
resources/research-engine/academic-pipeline/references/integrity_review_protocol.mdmarkdown7.1 KB
resources/research-engine/academic-pipeline/references/literature_corpus_consumers.mdmarkdown11.7 KB
resources/research-engine/academic-pipeline/references/mode_advisor.mdmarkdown7.8 KB
resources/research-engine/academic-pipeline/references/passport_as_reset_boundary.mdmarkdown18.6 KB
resources/research-engine/academic-pipeline/references/pipeline_state_machine.mdmarkdown28.3 KB
resources/research-engine/academic-pipeline/references/plagiarism_detection_protocol.mdmarkdown13.5 KB
resources/research-engine/academic-pipeline/references/process_summary_protocol.mdmarkdown15.8 KB
resources/research-engine/academic-pipeline/references/progress_dashboard_template.mdmarkdown1.5 KB
resources/research-engine/academic-pipeline/references/reinforcement_content.mdmarkdown1.4 KB
resources/research-engine/academic-pipeline/references/reproducibility_audit.mdmarkdown2.6 KB
resources/research-engine/academic-pipeline/references/score_trajectory_protocol.mdmarkdown3.8 KB
resources/research-engine/academic-pipeline/references/team_collaboration_protocol.mdmarkdown10.1 KB
resources/research-engine/academic-pipeline/references/two_stage_review_protocol.mdmarkdown3.0 KB
resources/research-engine/academic-pipeline/templates/pipeline_status_template.mdmarkdown3.7 KB
resources/research-engine/agents/report_compiler_agent.mdmarkdown26.0 KB
resources/research-engine/agents/research_architect_agent.mdmarkdown25.3 KB
resources/research-engine/agents/screening_reviewer_agent.mdmarkdown4.8 KB
resources/research-engine/agents/synthesis_agent.mdmarkdown28.2 KB
resources/research-engine/commands/ars-3w.mdmarkdown1.1 KB
resources/research-engine/commands/ars-abstract.mdmarkdown1.2 KB
resources/research-engine/commands/ars-cache-invalidate.mdmarkdown1.8 KB
resources/research-engine/commands/ars-citation-check.mdmarkdown971 B
resources/research-engine/commands/ars-disclosure.mdmarkdown1.5 KB
resources/research-engine/commands/ars-format-convert.mdmarkdown1.0 KB
resources/research-engine/commands/ars-full.mdmarkdown1.1 KB
resources/research-engine/commands/ars-lit-review.mdmarkdown1.3 KB
resources/research-engine/commands/ars-mark-read.mdmarkdown2.4 KB
resources/research-engine/commands/ars-outline.mdmarkdown953 B
resources/research-engine/commands/ars-plan.mdmarkdown965 B
resources/research-engine/commands/ars-rebuttal-audit.mdmarkdown1.3 KB
resources/research-engine/commands/ars-reviewer.mdmarkdown1.1 KB
resources/research-engine/commands/ars-revision-coach.mdmarkdown1.5 KB
resources/research-engine/commands/ars-revision.mdmarkdown944 B
resources/research-engine/commands/ars-unmark-read.mdmarkdown1.3 KB
resources/research-engine/deep-research/WORKFLOW.mdmarkdown51.3 KB
resources/research-engine/deep-research/agents/bibliography_agent.mdmarkdown34.4 KB
resources/research-engine/deep-research/agents/devils_advocate_agent.mdmarkdown10.9 KB
resources/research-engine/deep-research/agents/editor_in_chief_agent.mdmarkdown7.9 KB
resources/research-engine/deep-research/agents/ethics_review_agent.mdmarkdown21.7 KB
resources/research-engine/deep-research/agents/meta_analysis_agent.mdmarkdown16.3 KB
resources/research-engine/deep-research/agents/monitoring_agent.mdmarkdown9.2 KB
resources/research-engine/deep-research/agents/report_compiler_agent.mdmarkdown26.0 KB
resources/research-engine/deep-research/agents/research_architect_agent.mdmarkdown25.3 KB
resources/research-engine/deep-research/agents/research_question_agent.mdmarkdown10.1 KB
resources/research-engine/deep-research/agents/risk_of_bias_agent.mdmarkdown12.2 KB
resources/research-engine/deep-research/agents/socratic_mentor_agent.mdmarkdown52.3 KB
resources/research-engine/deep-research/agents/source_verification_agent.mdmarkdown10.3 KB
resources/research-engine/deep-research/agents/synthesis_agent.mdmarkdown28.2 KB
resources/research-engine/deep-research/agents/timeline_extraction_agent.mdmarkdown9.1 KB
resources/research-engine/deep-research/examples/exploratory_research.mdmarkdown6.9 KB
resources/research-engine/deep-research/examples/fact_check_mode.mdmarkdown12.9 KB
resources/research-engine/deep-research/examples/handoff_to_paper.mdmarkdown12.5 KB
resources/research-engine/deep-research/examples/idea_diversity_coverage_gap_advisory.mdmarkdown3.2 KB
resources/research-engine/deep-research/examples/policy_analysis.mdmarkdown9.4 KB
resources/research-engine/deep-research/examples/review_mode.mdmarkdown17.3 KB
resources/research-engine/deep-research/examples/socratic_guided_research.mdmarkdown22.9 KB
resources/research-engine/deep-research/examples/systematic_review.mdmarkdown7.7 KB
resources/research-engine/deep-research/references/apa7_style_guide.mdmarkdown4.6 KB
resources/research-engine/deep-research/references/argumentation_reasoning_framework.mdmarkdown3.9 KB
resources/research-engine/deep-research/references/arxiv_api_protocol.mdmarkdown6.5 KB
resources/research-engine/deep-research/references/changelog.mdmarkdown2.3 KB
resources/research-engine/deep-research/references/chinese_literature_api_protocol.mdmarkdown34.4 KB
resources/research-engine/deep-research/references/cross_agent_quality_definitions.mdmarkdown1.4 KB
resources/research-engine/deep-research/references/crossref_api_protocol.mdmarkdown5.0 KB
resources/research-engine/deep-research/references/equator_reporting_guidelines.mdmarkdown41.8 KB

resources/research-engine/academic-pipeline/references/claim_audit_calibration_protocol.md

View on GitHub
# Claim-Faithfulness Audit Calibration Protocol

**Status**: v3.8
**Parent agent**: `claim_ref_alignment_audit_agent`
**Mode name**: `calibration`
**Purpose**: Measure the audit agent's own false-negative rate (FNR) and false-positive rate (FPR) on alignment judgments (SUPPORTED / UNSUPPORTED / AMBIGUOUS / RETRIEVAL_FAILED) AND negative-constraint judgments (VIOLATED / NOT_VIOLATED) against a synthetic gold set, then gate CI on FNR < 0.15 AND FPR < 0.10.

This protocol is modeled on `academic-paper-reviewer/references/calibration_mode_protocol.md`. The reviewer mode measures editorial decision accuracy on whole papers; this mode measures claim-to-source alignment accuracy on per-citation tuples. Both share the same FNR / FPR / threshold-gate vocabulary; the difference is unit of analysis.

---

## Why this mode exists

The audit agent emits HIGH-WARN gate-refuse annotations on UNSUPPORTED rows and VIOLATED constraint matches. Those signals are only useful if the underlying judge is reliably distinguishing supported from unsupported claims. Without measurement, a judge that systematically over-flags AMBIGUOUS cases as UNSUPPORTED looks identical at the row level to a judge that nails the discrimination — but the former produces a flood of false HIGH-WARN refusals that erodes operator trust and eventually gets the agent ignored.

Calibration mode closes the measurability gap. It does NOT try to make the judge perfect; it makes the judge's error profile legible BEFORE the agent ships to a new operator's pipeline.

---

## Inputs

1. **Gold set**: 25 tuples (17 alignment incl. 5 #213 partial-support fixtures + 8 constraint per the canonical shipped fixture at `scripts/fixtures/claim_audit_calibration/gold_set.json`). Per-tuple shape and the four ingestion-gate rules (alignment-judgment coverage, ≥3 NOT_VIOLATED floor, required constraint fields, tuple_kind validity) are spelled out in **Gold tuple schema** below + spec §7.7 rules (a)-(d). The shipped fixture is the canonical example.

2. **Judge function**: a callable matching the v3.8 judge interface (`claim_text`, `retrieved_excerpt`, `anchor_kind`, `anchor_value`, `active_constraints`, `judge_model` kwargs). The shipped test fixture pairs the gold set with a perfect-judge stub so T-C1 / T-C2 / T-C3 exercise the calibration tooling end-to-end without requiring a live LLM call. Production deployment plugs a real judge_fn in place of the stub.

3. **Thresholds** (optional): `{"FNR": 0.15, "FPR": 0.10}` per spec §7.7 + §9 acceptance. Tightening these is a spec bump; loosening them is not permitted without a corresponding judge-quality-improvement justification recorded in the calibration report's `thresholds` block.

---

## Gold tuple schema

Two discriminated `tuple_kind` shapes, validated at ingestion by `validate_gold_set` (raises `GoldSetValidationError` on the first rule violation).

### Alignment tuple

```jsonc
{
  "tuple_kind": "alignment",
  "claim_text": "<the prose claim under test>",
  "ref_text_excerpt": "<source-passage excerpt the judge will evaluate against; null for RETRIEVAL_FAILED tuples>",
  "anchor": {"kind": "page" | "section" | "quote" | "paragraph" | "none", "value": "<...>"},
  "expected_judgment": "SUPPORTED" | "UNSUPPORTED" | "AMBIGUOUS" | "RETRIEVAL_FAILED"
}
```

Alignment tuples MUST NOT carry any of: `constraint_under_test_id`, `constraint_under_test_rule_text`, `manifest_fixture_path` (rule (b)). Mixing constraint fields onto an alignment tuple corrupts the per-class confusion matrix because the judge call shape differs between the two kinds.

### Constraint tuple

```jsonc
{
  "tuple_kind": "constraint",
  "claim_text": "<the prose claim under test>",
  "ref_text_excerpt": null,
  "anchor": {"kind": "page" | "section" | ..., "value": "<...>"},
  "expected_judgment": "VIOLATED" | "NOT_VIOLATED",
  "constraint_under_test_id": "MNC-N" | "NC-CN-M",
  "constraint_under_test_rule_text": "<inline rule text>",     // EITHER this
  "manifest_fixture_path": "scripts/fixtures/.../mnc.json"     // OR this
}
```

Both rule sources are accepted by `validate_gold_set` because manifest-bound constraints sometimes carry domain-specific rule text that authors want versioned alongside the manifest. **The v3.8.0 calibration runner supports only the inline `constraint_under_test_rule_text` form.** A manifest-only tuple validates clean per rule (c) but `run_calibration` raises `NotImplementedError` at run time rather than reach the judge with an empty rule (R2 codex P1 closure on the silent-skip risk that T-C3 is meant to prevent). The manifest-fixture resolver is a post-v3.8 deliverable; until it ships, author calibration fixtures with inline rule text.

---

## Process

### Phase 0: Intake

1. Call `validate_gold_set(tuples)`. Any of the four spec §7.7 rules (a)/(b)/(c)/(d) failing raises `GoldSetValidationError` with the offending tuple index and the rule name. Do NOT catch the exception — fix the gold set.

### Phase 1: Per-tuple judge invocation

For each tuple, the calibration runner calls `judge_fn(...)` with kwargs matching the v3.8 judge interface:
- Alignment tuples (all four `expected_judgment` values including `RETRIEVAL_FAILED`) → `retrieved_excerpt = tuple.ref_text_excerpt` (may be `null` for `RETRIEVAL_FAILED` tuples), `active_constraints=[]`. The runner does NOT pre-filter `RETRIEVAL_FAILED` tuples — the judge stub is responsible for returning the matching label.
- Constraint tuples → `retrieved_excerpt = tuple.ref_text_excerpt` (typically `null`), `active_constraints=[{"constraint_id": ..., "rule": ..., "scope": "MNC" | "NC"}]`. Scope is derived from `constraint_under_test_id`: `MNC-N` → `"MNC"`, `NC-CN-M` → `"NC"`. Malformed ids are rejected by `_derive_constraint_scope` with a diagnostic naming the expected shape.

No ensembling at this layer. Spec §7.7 does not require N-run majority voting (reviewer mode does, but reviewer-paper unit-of-analysis is heavier and benefits more from variance reduction; per-tuple alignment is a simpler call). Re-running the same tuple is the operator's responsibility if a high-variance judge_model warrants it.

### Phase 2: Confusion-matrix accumulation

The runner accumulates two confusion matrices:

- **Aggregate** (the rates that T-C1 gates against): a tuple contributes a FN when `expected_judgment ≠ actual judgment`. For alignment, every mismatched tuple counts as both FN (for the expected class) and FP (for the wrongly-picked class), so aggregate FNR and FPR are symmetric. For constraint tuples, VIOLATED is the positive class (gate-refuse signal); the matrix is the standard binary form.
- **Per-class one-vs-rest** (the report block that T-C2 checks): for each of `SUPPORTED`, `UNSUPPORTED`, `AMBIGUOUS`, `violated_constraint`, the runner computes FNR / FPR with that class as the positive label. `n_positive` / `n_negative` denominators are surfaced so a "0.0 FNR on n_positive=0" entry is distinguishable from "0.0 FNR on n_positive=8".

The fourth alignment class `RETRIEVAL_FAILED` is intentionally NOT surfaced in `per_class`: the pipeline sets it pre-judge at operational deployment, so judge-quality FNR/FPR against this label is uninformative for tooling validation. To keep the per-tuple call shape uniform, the runner still passes `RETRIEVAL_FAILED` tuples to `judge_fn` (no pre-filter) — the stub or real judge echoes the expected label, and the tuple contributes to aggregate FNR/FPR via the non-`per_class` aggregate path.

### Phase 3: Threshold check

If `report["FNR"] >= thresholds["FNR"]` OR `report["FPR"] >= thresholds["FPR"]`, the test (T-C1) fails. The report's `thresholds` block echoes the active gate values so CI failure messages can distinguish:

- A regression (judge quality degraded — same gold set, higher FNR/FPR than the previous run).
- A spec bump (tighter thresholds — same judge, but T-C1 now demands more).

Both surface as the same CI failure; the report block makes the diagnosis explicit.

### Phase 4: Report shape

```jsonc
{
  "FNR": 0.0,
  "FPR": 0.0,
  "per_class": {
    "SUPPORTED":           {"FNR": 0.0, "FPR": 0.0, "n_positive": 5, "n_negative": 12},
    "UNSUPPORTED":         {"FNR": 0.0, "FPR": 0.0, "n_positive": 8, "n_negative": 9},
    "AMBIGUOUS":           {"FNR": 0.0, "FPR": 0.0, "n_positive": 3, "n_negative": 14},
    "violated_constraint": {"FNR": 0.0, "FPR": 0.0, "n_positive": 5, "n_negative": 3}
  },
  "partial_support": {"miss_rate": 0.0, "n_partial": 5},
  "thresholds": {"FNR": 0.15, "FPR": 0.10},
  "n_total": 25,
  "n_alignment": 17,
  "n_constraint": 8
}
```

The shape is the canonical contract `scripts/test_claim_audit_calibration.py` (T-C1 / T-C2) pins against. Adding fields is non-breaking; removing or renaming is a spec bump.

**Partial-support subset (`partial_support`, #213).** Gold tuples tagged `expected_prompt_verdict=PARTIAL` carry `expected_judgment=UNSUPPORTED` (the B1 normalization), so a judge that **stops decomposing and emits a bare `UNSUPPORTED`** matches the aggregate gate and hides the exact regression #213 closes. The subset metric counts a partial fixture as passed ONLY when the judge emits `UNSUPPORTED` AND a well-formed true-partial `sub_claim_breakdown` (≥2 items, ≥1 SUPPORTED AND ≥1 non-SUPPORTED — the INV-19 definition). `miss_rate = misses / n_partial`; a bare-`UNSUPPORTED` judge registers `miss_rate=1.0` here while the aggregate FNR stays green. The block is additive — a gold set with no partial fixtures yields `{"miss_rate": 0.0, "n_partial": 0}`.

---

## Failure cases this mode does NOT fix

Calibration reports the judge's error profile on a **specific** gold set in a **specific** domain. It does not:

- Predict performance on tuples outside the gold-set distribution (the canonical fixture is mid-domain synthetic; ML / clinical / qualitative judges should run domain-specific gold sets).
- Detect rubric-discrimination-power problems on the LLM-as-judge side — that's the `#89 / gold fixtures` work tracked separately (per spec §2 out-of-scope).
- Replace the post-calibration ramp-on plan recorded in `academic-pipeline/WORKFLOW.md` mode flags. v3.8.0 ships the audit agent dispatch as opt-in default-OFF; T-C1 passing on the canonical gold set is necessary but not sufficient evidence for default-ON.

If a deploying operator brings a gold set that is itself biased (all tuples from one venue, all post-2024, all heavily-redacted), calibration reports a biased profile. Emit a warning during intake if the gold set looks clustered — this protocol does not currently auto-detect clustering, but the operator's pre-deployment review should.

**Partial-evidence regression IS caught (#213).** One failure the aggregate gate alone would NOT catch: a judge that regresses to emitting bare `UNSUPPORTED` on compound claims instead of decomposing them. Because partial fixtures carry `expected_judgment=UNSUPPORTED`, the aggregate FNR stays green on that regression. The `partial_support` subset metric exists precisely to surface it — `miss_rate > 0` means the judge stopped producing inspectable `sub_claim_breakdown` decompositions even though it got the headline label right. Treat a non-zero `partial_support.miss_rate` as a judge-quality regression on the §F.3.2 partial-evidence trap, distinct from an aggregate FNR/FPR threshold breach.

---

## Integration with existing modes

| Mode | Interaction with calibration |
|---|---|
| Stage 4 → 5 audit dispatch | Calibration runs offline; results inform the `ARS_CLAIM_AUDIT` ramp-on decision. |
| Cite-Time Provenance Finalizer | No direct coupling — calibration evaluates the judge in isolation; the finalizer's matrix is downstream. |
| `/ars-mark-read` clearance | Calibration does not interact with mark-read state; HIGH-WARN-CLAIM-NOT-SUPPORTED stays non-clearable regardless of calibration outcome. |
| Reviewer `calibration` mode | Sibling, not coupled. Both compute FNR/FPR but on different units (papers vs claim-tuples); both can run in the same session. |

---

## Resolved design decisions (2026-05-15, per spec §10 OQs)

- **Activation**: opt-in. The audit agent ships with default thresholds and the canonical fixture; operators run `scripts/test_claim_audit_calibration` as a CI gate. Re-calibration with a domain-specific gold set is the operator's call, not auto-triggered.
- **Threshold values**: FNR < 0.15 + FPR < 0.10. Tightened from the reviewer-mode 0.17 / 0.50 Lu 2026 reference points because the audit unit (per-claim) is simpler than the reviewer unit (whole paper) — the gate should track the stricter end of plausible LLM-as-judge accuracy.
- **Ensembling**: not in v3.8.0. Per-tuple alignment calls are short and the judge's response shape is constrained; majority-vote ensembling adds cost without obvious accuracy gain for this unit of analysis. Re-evaluate if calibration evidence in v3.8.x shows high variance.
- **Cross-model verification**: out of scope for the calibration runner — `ARS_CROSS_MODEL` interacts with the audit agent dispatch path, not the calibration script. An operator wanting cross-model calibration runs the runner twice with different `judge_model` settings.

---

## Running locally

```bash
PYTHONPATH=. python3 -m unittest scripts.test_claim_audit_calibration -v
```

The CI-equivalent invocation lives in `.github/workflows/spec-consistency.yml` (the v3.8 `#103` unittest step). Both paths exercise T-C1 (threshold gates), T-C2 (per-class FNR/FPR reporting), and T-C3 (gold-set shape integrity).

---

## References

- Spec: `docs/design/2026-05-15-issue-103-claim-alignment-audit-spec.md` §7.7 (test contract) + §1 deliverable 7 (this doc) + §9 (acceptance criteria) + §13 step 10 (implementation order).
- Issue: [academic-research-skills #103](https://github.com/Imbad0202/academic-research-skills/issues/103) acceptance criterion (FNR < 0.15 + FPR < 0.10 gates).
- Parent agent: `academic-pipeline/agents/claim_ref_alignment_audit_agent.md` — dispatches calibration mode and consumes the report's thresholds block. The agent prompt and this doc form a two-way pair following the v3.6.5 protocol-doc convention (`literature_corpus_consumers.md` ↔ `bibliography_agent.md`).
- Reviewer calibration baseline: `academic-paper-reviewer/references/calibration_mode_protocol.md` (same vocabulary, different unit of analysis).
- Lu, C. et al. (2026). *Nature* 651, 914-919 — Table 1 reference rates for LLM-vs-human agreement on whole-paper decisions; this protocol uses tighter thresholds for the simpler per-claim unit.
add-product-e2e-evalpaperclipai/paperclipAdd or extend a Paperclip full-stack runner E2E workflow, fixture, matcher, or report evidence path for local or Daytona execution.Stars: 115kUpdated Claude CodeCodexVerifiedInstall
add-runner-evalpaperclipai/paperclipAdd or extend a Paperclip Runner protocol evaluation definition, roster, assertion, or report fixture with provenance and narrow validation.Stars: 115kUpdated Claude CodeCodexVerifiedInstall
diagnose-why-work-stoppedpaperclipai/paperclipCompatibility entry point for failed, stalled, looping, or over-recovered Paperclip work. Use fix-failed-paperclip-run for the full diagnosis, regression, repair, real verification, and green-PR workflow; honor an explicitly requested diagnosis-only scope.Stars: 115kUpdated Claude CodeCodexVerifiedInstall
fix-failed-paperclip-runpaperclipai/paperclipDiagnose and fix failed, stuck, slow, or incorrectly blocked Paperclip tasks and runs. Use with a task/run URL or failure report to trace the root cause, prove a regression red then green, check second-order effects, verify the real workflow, and deliver a PR with passing checks.Stars: 115kUpdated Claude CodeCodexVerifiedInstall