Skip to content

YCC-Lover/research-starter-skills / skills/research-starter-paper

research-starter-paper

Unverified

科研学习与论文写作总入口,协调六个模块。帮助不知道提供什么材料的新手逐步产出、自动保存和恢复项目进度,也用于跨阶段研究任务;区分学习、执行和只读检查。

FileKindSize
SKILL.mdskill md8.8 KB
agents/openai.yamlconfig366 B
assets/evidence-ledger.tsvother106 B
assets/manuscript-plan.mdmarkdown423 B
assets/project-state.mdmarkdown928 B
assets/requirements-research.txtmarkdown263 B
assets/task-brief.mdmarkdown429 B
references/advanced-capabilities.jsonconfig8.5 KB
references/advanced-research.mdmarkdown8.5 KB
references/discipline-fit.mdmarkdown2.3 KB
references/evidence-and-figures.mdmarkdown2.8 KB
references/guided-start.mdmarkdown7.2 KB
references/handoff.mdmarkdown1.9 KB
references/learning.mdmarkdown3.2 KB
references/paper-sections.mdmarkdown5.6 KB
references/research-engine-notice.mdmarkdown1.6 KB
references/research-engine-patches.jsonconfig3.4 KB
references/research-practice.mdmarkdown3.2 KB
references/research.mdmarkdown3.2 KB
references/revision.mdmarkdown2.1 KB
references/runtime-helpers.mdmarkdown6.0 KB
references/source-manifest.jsonconfig34.8 KB
references/sources.mdmarkdown7.4 KB
references/task-brief.mdmarkdown1.6 KB
resources/research-engine/LICENSEother19.1 KB
resources/research-engine/MODE_REGISTRY.mdmarkdown6.7 KB
resources/research-engine/NOTICE.mdmarkdown1.0 KB
resources/research-engine/THIRD_PARTY.mdmarkdown5.0 KB
resources/research-engine/academic-paper-reviewer/WORKFLOW.mdmarkdown43.9 KB
resources/research-engine/academic-paper-reviewer/agents/devils_advocate_reviewer_agent.mdmarkdown42.0 KB
resources/research-engine/academic-paper-reviewer/agents/domain_reviewer_agent.mdmarkdown32.4 KB
resources/research-engine/academic-paper-reviewer/agents/editorial_synthesizer_agent.mdmarkdown41.8 KB
resources/research-engine/academic-paper-reviewer/agents/eic_agent.mdmarkdown28.1 KB
resources/research-engine/academic-paper-reviewer/agents/field_analyst_agent.mdmarkdown11.2 KB
resources/research-engine/academic-paper-reviewer/agents/methodology_reviewer_agent.mdmarkdown44.0 KB
resources/research-engine/academic-paper-reviewer/agents/perspective_reviewer_agent.mdmarkdown32.7 KB
resources/research-engine/academic-paper-reviewer/examples/hei_paper_review_example.mdmarkdown33.8 KB
resources/research-engine/academic-paper-reviewer/examples/interdisciplinary_review_example.mdmarkdown27.6 KB
resources/research-engine/academic-paper-reviewer/examples/subclaim_decomposition_example.mdmarkdown8.3 KB
resources/research-engine/academic-paper-reviewer/references/calibration_mode_protocol.mdmarkdown15.5 KB
resources/research-engine/academic-paper-reviewer/references/changelog.mdmarkdown1.4 KB
resources/research-engine/academic-paper-reviewer/references/editorial_decision_standards.mdmarkdown12.8 KB
resources/research-engine/academic-paper-reviewer/references/guided_mode_protocol.mdmarkdown2.0 KB
resources/research-engine/academic-paper-reviewer/references/integration_guide.mdmarkdown680 B
resources/research-engine/academic-paper-reviewer/references/quality_rubrics.mdmarkdown6.3 KB
resources/research-engine/academic-paper-reviewer/references/re_review_mode_protocol.mdmarkdown51.9 KB
resources/research-engine/academic-paper-reviewer/references/review_criteria_framework.mdmarkdown6.2 KB
resources/research-engine/academic-paper-reviewer/references/review_panel_provenance_protocol.mdmarkdown9.4 KB
resources/research-engine/academic-paper-reviewer/references/review_quality_thinking.mdmarkdown3.3 KB
resources/research-engine/academic-paper-reviewer/references/reviewer_sprint_prompt_source.mdmarkdown47.9 KB
resources/research-engine/academic-paper-reviewer/references/sprint_contract_protocol.mdmarkdown32.1 KB
resources/research-engine/academic-paper-reviewer/references/statistical_reporting_standards.mdmarkdown29.8 KB
resources/research-engine/academic-paper-reviewer/references/top_journals_by_field.mdmarkdown12.6 KB
resources/research-engine/academic-paper-reviewer/templates/editorial_decision_template.mdmarkdown11.0 KB
resources/research-engine/academic-paper-reviewer/templates/peer_review_report_template.mdmarkdown14.8 KB
resources/research-engine/academic-paper-reviewer/templates/revision_response_template.mdmarkdown7.4 KB
resources/research-engine/academic-paper/WORKFLOW.mdmarkdown63.8 KB
resources/research-engine/academic-paper/agents/abstract_bilingual_agent.mdmarkdown12.0 KB
resources/research-engine/academic-paper/agents/argument_builder_agent.mdmarkdown10.9 KB
resources/research-engine/academic-paper/agents/citation_compliance_agent.mdmarkdown23.5 KB
resources/research-engine/academic-paper/agents/draft_writer_agent.mdmarkdown56.1 KB
resources/research-engine/academic-paper/agents/formatter_agent.mdmarkdown63.8 KB
resources/research-engine/academic-paper/agents/intake_agent.mdmarkdown25.5 KB
resources/research-engine/academic-paper/agents/literature_strategist_agent.mdmarkdown46.5 KB
resources/research-engine/academic-paper/agents/peer_reviewer_agent.mdmarkdown29.5 KB
resources/research-engine/academic-paper/agents/revision_coach_agent.mdmarkdown20.0 KB
resources/research-engine/academic-paper/agents/socratic_mentor_agent.mdmarkdown27.9 KB
resources/research-engine/academic-paper/agents/structure_architect_agent.mdmarkdown18.9 KB
resources/research-engine/academic-paper/agents/visualization_agent.mdmarkdown17.2 KB
resources/research-engine/academic-paper/examples/chinese_paper_example.mdmarkdown21.1 KB
resources/research-engine/academic-paper/examples/clinical_citation_verification_checklist.mdmarkdown4.4 KB
resources/research-engine/academic-paper/examples/clinical_epistemic_status_example.mdmarkdown6.6 KB
resources/research-engine/academic-paper/examples/commitment_ledger_example.mdmarkdown8.1 KB
resources/research-engine/academic-paper/examples/imrad_hei_example.mdmarkdown23.3 KB
resources/research-engine/academic-paper/examples/literature_review_example.mdmarkdown30.8 KB
resources/research-engine/academic-paper/examples/plan_mode_guided_writing.mdmarkdown34.3 KB
resources/research-engine/academic-paper/examples/revision_mode_example.mdmarkdown20.1 KB
resources/research-engine/academic-paper/examples/revision_recovery_example.mdmarkdown41.2 KB
resources/research-engine/academic-paper/examples/version_family_reconciliation_example.mdmarkdown3.8 KB
resources/research-engine/academic-paper/references/abstract_writing_guide.mdmarkdown12.5 KB
resources/research-engine/academic-paper/references/academic_writing_style.mdmarkdown7.6 KB
resources/research-engine/academic-paper/references/anti_leakage_protocol.mdmarkdown4.4 KB
resources/research-engine/academic-paper/references/apa7_chinese_citation_guide.mdmarkdown14.0 KB
resources/research-engine/academic-paper/references/apa7_extended_guide.mdmarkdown6.6 KB
resources/research-engine/academic-paper/references/changelog.mdmarkdown2.8 KB
resources/research-engine/academic-paper/references/citation_format_switcher.mdmarkdown7.4 KB
resources/research-engine/academic-paper/references/committee_correspondence_protocol.mdmarkdown6.3 KB
resources/research-engine/academic-paper/references/credit_authorship_guide.mdmarkdown13.7 KB
resources/research-engine/academic-paper/references/disclosure_mode_protocol.mdmarkdown53.9 KB
resources/research-engine/academic-paper/references/domain_evidence_profiles.mdmarkdown4.2 KB
resources/research-engine/academic-paper/references/failure_paths.mdmarkdown14.2 KB
resources/research-engine/academic-paper/references/funding_statement_guide.mdmarkdown11.9 KB
resources/research-engine/academic-paper/references/hei_domain_glossary.mdmarkdown6.8 KB
resources/research-engine/academic-paper/references/intro_title_rhetoric_guide.mdmarkdown8.7 KB
resources/research-engine/academic-paper/references/journal_submission_guide.mdmarkdown8.7 KB
resources/research-engine/academic-paper/references/latex_template_reference.mdmarkdown9.6 KB
resources/research-engine/academic-paper/references/mode_selection_guide.mdmarkdown17.5 KB
resources/research-engine/academic-paper/references/paper_structure_patterns.mdmarkdown8.4 KB
resources/research-engine/academic-paper/references/plan_mode_protocol.mdmarkdown6.7 KB
resources/research-engine/academic-paper/references/policy_anchor_disclosure_protocol.mdmarkdown21.0 KB
resources/research-engine/academic-paper/references/policy_anchor_table.mdmarkdown21.2 KB
resources/research-engine/academic-paper/references/revision_patch_protocol.mdmarkdown12.9 KB
resources/research-engine/academic-paper/references/statistical_visualization_standards.mdmarkdown23.7 KB
resources/research-engine/academic-paper/references/venue_disclosure_policies.mdmarkdown38.4 KB
resources/research-engine/academic-paper/references/vlm_figure_verification.mdmarkdown9.1 KB
resources/research-engine/academic-paper/references/workflow_phase_details.mdmarkdown4.7 KB
resources/research-engine/academic-paper/references/writing_judgment_framework.mdmarkdown3.3 KB
resources/research-engine/academic-paper/references/writing_quality_check.mdmarkdown13.0 KB
resources/research-engine/academic-paper/templates/bilingual_abstract_template.mdmarkdown4.7 KB
resources/research-engine/academic-paper/templates/case_study_template.mdmarkdown2.8 KB
resources/research-engine/academic-paper/templates/conference_paper_template.mdmarkdown2.3 KB
resources/research-engine/academic-paper/templates/credit_statement_template.mdmarkdown4.4 KB
resources/research-engine/academic-paper/templates/funding_statement_template.mdmarkdown8.9 KB
resources/research-engine/academic-paper/templates/imrad_template.mdmarkdown4.4 KB
resources/research-engine/academic-paper/templates/latex_article_template.texother4.7 KB
resources/research-engine/academic-paper/templates/literature_review_template.mdmarkdown2.8 KB
resources/research-engine/academic-paper/templates/policy_brief_template.mdmarkdown3.6 KB
resources/research-engine/academic-paper/templates/revision_tracking_template.mdmarkdown10.2 KB
resources/research-engine/academic-paper/templates/theoretical_paper_template.mdmarkdown2.8 KB
resources/research-engine/academic-pipeline/WORKFLOW.mdmarkdown58.0 KB
resources/research-engine/academic-pipeline/agents/claim_ref_alignment_audit_agent.mdmarkdown44.3 KB
resources/research-engine/academic-pipeline/agents/collaboration_depth_agent.mdmarkdown9.8 KB
resources/research-engine/academic-pipeline/agents/integrity_verification_agent.mdmarkdown70.7 KB
resources/research-engine/academic-pipeline/agents/pipeline_orchestrator_agent.mdmarkdown173.8 KB
resources/research-engine/academic-pipeline/agents/state_tracker_agent.mdmarkdown26.8 KB
resources/research-engine/academic-pipeline/examples/full_pipeline_example.mdmarkdown17.7 KB
resources/research-engine/academic-pipeline/examples/integrity_failure_recovery.mdmarkdown26.8 KB
resources/research-engine/academic-pipeline/examples/mid_entry_example.mdmarkdown13.4 KB
resources/research-engine/academic-pipeline/references/adapters/.gitkeepother0 B
resources/research-engine/academic-pipeline/references/adapters/overview.mdmarkdown10.0 KB
resources/research-engine/academic-pipeline/references/ai_research_failure_modes.mdmarkdown15.6 KB
resources/research-engine/academic-pipeline/references/changelog.mdmarkdown5.8 KB
resources/research-engine/academic-pipeline/references/claim_audit_calibration_protocol.mdmarkdown14.2 KB
resources/research-engine/academic-pipeline/references/claim_verification_protocol.mdmarkdown27.2 KB
resources/research-engine/academic-pipeline/references/external_review_protocol.mdmarkdown6.2 KB
resources/research-engine/academic-pipeline/references/integrity_review_protocol.mdmarkdown7.1 KB
resources/research-engine/academic-pipeline/references/literature_corpus_consumers.mdmarkdown11.7 KB
resources/research-engine/academic-pipeline/references/mode_advisor.mdmarkdown7.8 KB
resources/research-engine/academic-pipeline/references/passport_as_reset_boundary.mdmarkdown18.6 KB
resources/research-engine/academic-pipeline/references/pipeline_state_machine.mdmarkdown28.3 KB
resources/research-engine/academic-pipeline/references/plagiarism_detection_protocol.mdmarkdown13.5 KB
resources/research-engine/academic-pipeline/references/process_summary_protocol.mdmarkdown15.8 KB
resources/research-engine/academic-pipeline/references/progress_dashboard_template.mdmarkdown1.5 KB
resources/research-engine/academic-pipeline/references/reinforcement_content.mdmarkdown1.4 KB
resources/research-engine/academic-pipeline/references/reproducibility_audit.mdmarkdown2.6 KB
resources/research-engine/academic-pipeline/references/score_trajectory_protocol.mdmarkdown3.8 KB
resources/research-engine/academic-pipeline/references/team_collaboration_protocol.mdmarkdown10.1 KB
resources/research-engine/academic-pipeline/references/two_stage_review_protocol.mdmarkdown3.0 KB
resources/research-engine/academic-pipeline/templates/pipeline_status_template.mdmarkdown3.7 KB
resources/research-engine/agents/report_compiler_agent.mdmarkdown26.0 KB
resources/research-engine/agents/research_architect_agent.mdmarkdown25.3 KB
resources/research-engine/agents/screening_reviewer_agent.mdmarkdown4.8 KB
resources/research-engine/agents/synthesis_agent.mdmarkdown28.2 KB
resources/research-engine/commands/ars-3w.mdmarkdown1.1 KB
resources/research-engine/commands/ars-abstract.mdmarkdown1.2 KB
resources/research-engine/commands/ars-cache-invalidate.mdmarkdown1.8 KB
resources/research-engine/commands/ars-citation-check.mdmarkdown971 B
resources/research-engine/commands/ars-disclosure.mdmarkdown1.5 KB
resources/research-engine/commands/ars-format-convert.mdmarkdown1.0 KB
resources/research-engine/commands/ars-full.mdmarkdown1.1 KB
resources/research-engine/commands/ars-lit-review.mdmarkdown1.3 KB
resources/research-engine/commands/ars-mark-read.mdmarkdown2.4 KB
resources/research-engine/commands/ars-outline.mdmarkdown953 B
resources/research-engine/commands/ars-plan.mdmarkdown965 B
resources/research-engine/commands/ars-rebuttal-audit.mdmarkdown1.3 KB
resources/research-engine/commands/ars-reviewer.mdmarkdown1.1 KB
resources/research-engine/commands/ars-revision-coach.mdmarkdown1.5 KB
resources/research-engine/commands/ars-revision.mdmarkdown944 B
resources/research-engine/commands/ars-unmark-read.mdmarkdown1.3 KB
resources/research-engine/deep-research/WORKFLOW.mdmarkdown51.3 KB
resources/research-engine/deep-research/agents/bibliography_agent.mdmarkdown34.4 KB
resources/research-engine/deep-research/agents/devils_advocate_agent.mdmarkdown10.9 KB
resources/research-engine/deep-research/agents/editor_in_chief_agent.mdmarkdown7.9 KB
resources/research-engine/deep-research/agents/ethics_review_agent.mdmarkdown21.7 KB
resources/research-engine/deep-research/agents/meta_analysis_agent.mdmarkdown16.3 KB
resources/research-engine/deep-research/agents/monitoring_agent.mdmarkdown9.2 KB
resources/research-engine/deep-research/agents/report_compiler_agent.mdmarkdown26.0 KB
resources/research-engine/deep-research/agents/research_architect_agent.mdmarkdown25.3 KB
resources/research-engine/deep-research/agents/research_question_agent.mdmarkdown10.1 KB
resources/research-engine/deep-research/agents/risk_of_bias_agent.mdmarkdown12.2 KB
resources/research-engine/deep-research/agents/socratic_mentor_agent.mdmarkdown52.3 KB
resources/research-engine/deep-research/agents/source_verification_agent.mdmarkdown10.3 KB
resources/research-engine/deep-research/agents/synthesis_agent.mdmarkdown28.2 KB
resources/research-engine/deep-research/agents/timeline_extraction_agent.mdmarkdown9.1 KB
resources/research-engine/deep-research/examples/exploratory_research.mdmarkdown6.9 KB
resources/research-engine/deep-research/examples/fact_check_mode.mdmarkdown12.9 KB
resources/research-engine/deep-research/examples/handoff_to_paper.mdmarkdown12.5 KB
resources/research-engine/deep-research/examples/idea_diversity_coverage_gap_advisory.mdmarkdown3.2 KB
resources/research-engine/deep-research/examples/policy_analysis.mdmarkdown9.4 KB
resources/research-engine/deep-research/examples/review_mode.mdmarkdown17.3 KB
resources/research-engine/deep-research/examples/socratic_guided_research.mdmarkdown22.9 KB
resources/research-engine/deep-research/examples/systematic_review.mdmarkdown7.7 KB
resources/research-engine/deep-research/references/apa7_style_guide.mdmarkdown4.6 KB
resources/research-engine/deep-research/references/argumentation_reasoning_framework.mdmarkdown3.9 KB
resources/research-engine/deep-research/references/arxiv_api_protocol.mdmarkdown6.5 KB
resources/research-engine/deep-research/references/changelog.mdmarkdown2.3 KB
resources/research-engine/deep-research/references/chinese_literature_api_protocol.mdmarkdown34.4 KB
resources/research-engine/deep-research/references/cross_agent_quality_definitions.mdmarkdown1.4 KB
resources/research-engine/deep-research/references/crossref_api_protocol.mdmarkdown5.0 KB
resources/research-engine/deep-research/references/equator_reporting_guidelines.mdmarkdown41.8 KB

resources/research-engine/academic-paper-reviewer/references/calibration_mode_protocol.md

View on GitHub
# Calibration Mode Protocol

**Parent skill**: `academic-paper-reviewer`
**Mode name**: `calibration`
**Purpose**: Measure a bounded decision-error profile against a user-adjudicated target set, or obtain a cheaper directional readout. This protocol never turns criterion judgements into an absolute quality score.

## Epistemic status

An ordinary review has `calibration_status: NOT_CALIBRATED`. Its categorical judgements are tied to named criteria and evidence in one manuscript; they do not establish a stable interval scale, paper ranking, acceptance probability, or cross-session reproducibility.

The full tier may produce a candidate empirical target profile only after completing the protocol against the current domain, article type, venue criteria, rubric version, review mode, and exact replay-derived `execution_topology_sha256`. `PROFILE_MEASURED` means only that errors were measured on that bounded target set. It is not a claim of universal calibration and must not be transported to a materially different target or review topology.

**Current application boundary:** candidate profile production and live-review
profile application are separate operations. Until a closed, hash-bound target
profile schema plus replay validator is shipped, reviewer seats and the current
Schema 6 package adapter emit `calibration_status: NOT_CALIBRATED`; a prose
profile ID or apparent topology match cannot upgrade them. The full-tier report
may describe its measured candidate profile, but must label it
`application_status: NOT_WIRED_TO_LIVE_REVIEW`.

The directional tier always reports `calibration_status: NOT_CALIBRATED`. Three single panels provide directional observations, not an error profile.

## Inputs

1. **Tier**: `full` by default. Use `directional` only when explicitly selected.
2. **Empirical target set**:
   - Full: 5–20 user-adjudicated papers, preferably 10–15, with at least one acceptable-side and one reject-side label.
   - Directional: exactly three papers—one `minor_revision`, one `major_revision`, and one extreme anchor (`accept` or `reject`).
3. **Per-paper fields**:
   - manuscript path or text;
   - adjudicated verdict: `accept`, `minor_revision`, `major_revision`, `reject`, or legacy `borderline` in the full tier only;
   - venue, domain, and article-type context;
   - optional `per_dimension_gold_judgements`, using the same criterion-bound categories as `quality_rubrics.md`, with the criterion source and adjudication rationale.
4. **Target-profile identity**: domain, article type, venue criteria/version, rubric version, review mode, target-set identifier, adjudication date, and one exact `execution_topology_sha256` derived from actual completed panel provenance. A prose configuration label or intended route is not a match key.
5. **Session persistence**: session-only. Do not cache manuscripts or profiles across sessions.

### Gold-label isolation

Gold verdicts, dimension judgements, rationales, and human assessments must not enter field-analyst, reviewer, or synthesizer context. Join them only after a panel verdict is frozen. The substrate plan must also be fixed without consulting gold material.

This isolation applies to transport as well as prompt construction: no gold
verdict, dimension judgement, rationale, human assessment, or target-set
outcome may enter a provider payload or determine the substrate plan. The join
is post-freeze only, and joined material cannot be fed back into another panel
within the same attempt.

## Process

### Phase 0: Intake

- Full tier: verify 5–20 papers and both sides of the binary decision boundary. Warn that small or clustered target sets yield uncertain, non-transportable estimates. Legacy `borderline` papers are reported separately and excluded from binary metrics.
- Directional tier: verify exactly the required three-paper composition, one run per paper, and no `borderline` item. State that the result remains `NOT_CALIBRATED`.
- Never infer the tier from paper count; ask when the user's selection is missing.

### Phase 1: Run panels

**Full tier.** Run 3 or 5 panel replicates per paper (default 5), then derive the panel verdict by majority vote. Within each replicate, the five seats must have distinct recorded invocation-context IDs. The current provenance artifact checks only that within-panel separation; it does not compare IDs against earlier replicates and therefore does not establish cross-replicate freshness or independent error processes. Preserve each seat's criterion-bound judgements and evidence; do not convert them to numbers or average them. Report exact-agreement and disagreement patterns for a dimension only when the matching gold judgement exists.

**Directional tier.** Run one panel per paper with distinct recorded invocation-context IDs among its five seats. Preserve the exact panel verdict and each scoring seat's categorical dimension judgements as emitted. Do not ensemble, manufacture variance, combine labels into a panel score, or rank the papers.

**Cross-model verification and actual provenance.** `ARS_CROSS_MODEL` is
default-on for calibration mode. Follow
`shared/cross_model_verification.md` § Calibration transport exception. Before
any provider call, run its closed calibration data-fence collision preflight
independently on the raw reviewer-configuration bytes and raw manuscript bytes.
A collision refuses the entire attempt before transport: do not send either
payload and do not escape, strip, rewrite, truncate, switch delimiters, or
silently fall back.

Create an `attempt_id`, lock one `substrate_plan` before any gold material is
consulted, and use the same transport and substrate for every paper and
replicate. For every completed panel, build and replay-validate
`review-panel-provenance/1.0` from actual seat executions. All result-producing
panels in one calibration attempt MUST have one identical
`execution_topology_sha256`; every artifact must derive `fresh_context: true`
under its fixed `within_panel_attempt_only` scope. The builder receives no
attempt-history ledger: even if invocation-context IDs differ within each
artifact, it cannot detect reuse across papers or replicates. Every candidate
profile and readout must disclose that cross-replicate freshness is unverified
and must not describe the repeated panels as independent. If consent, configuration, or
non-content transport preflight is unavailable before the attempt begins, lock
all seats to the primary family and execute the complete schedule on that
homogeneous plan.

A mid-attempt dispatch or substrate failure, topology mismatch, unknown
required provenance observation, within-panel context reuse, or invalid provenance
artifact invalidates the whole attempt. Every completed panel becomes
diagnostic-only and must not enter any aggregate. The only result-producing
retry uses a new `attempt_id`, an empty aggregate, and restarts at paper 1 /
replicate 1 on one homogeneous plan. Never resume the failed attempt, mix
transports or substrates, or emit a profile or directional readout from an
incomplete, mixed-substrate, mixed-topology, or provenance-unresolved attempt.

### Phase 2: Full-tier decision-error profile

Map adjudicated and panel verdicts as follows: Accept/Minor Revision are the acceptable side; Major Revision/Reject are the reject side. With positive meaning acceptable:

| Metric | Meaning | Reporting boundary |
|---|---|---|
| Balanced accuracy | Mean of sensitivity on the two sides | Point estimate plus bootstrap interval |
| FNR | Acceptable-side papers judged Major/Reject (over-harsh) | Point estimate plus bootstrap interval |
| FPR | Reject-side papers judged Accept/Minor (too lenient) | Point estimate plus bootstrap interval |
| Exact four-label agreement | Exact verdict matches | Count and share, with target-set size |

Do not report AUC: there is no continuous rubric score. Do not infer an ordinal paper ranking from the four editorial labels.

For each dimension with adjudicated gold judgements, report a categorical agreement table and `annotated_n=<n>/<N>, missing=<N-n>`. With no annotated gold for a dimension, report `NOT COMPUTABLE`. Do not assign distances between categories, average the labels, or claim a gold-set-wide dimension result from a subset.

### Phase 2.5: Minor/Major boundary (both tiers)

When both `minor_revision` and `major_revision` gold examples exist, report raw counts:

| Gold \ predicted side | Accept + Minor | Major + Reject |
|---|---:|---:|
| Minor Revision | stayed minor-side | harsh crossing |
| Major Revision | lenient crossing | stayed major-side |

If either side is absent in a full-tier target set, report `NOT ESTIMABLE — target set lacks both sides of the Minor/Major boundary`. Never render an all-zero table as evidence of no confusion.

### Phase 2.6: Directional reporting boundary

The directional readout may report only:

- each paper's exact gold verdict, exact panel verdict, per-seat categorical criterion judgements, and `lenient` / `exact` / `harsh` direction;
- raw direction counts;
- the raw Minor/Major boundary cells; and
- raw low/med/high severity-grounding-risk counts from Phase 3.5.

It must not report balanced accuracy, FNR, FPR, AUC, confidence intervals, stability, per-dimension error rates, numeric baseline comparisons, or any measured-profile claim.

### Phase 3.5: Severity-miscalibration measurement (#215)

Decision-level error does not capture findings whose severity relies on an asserted field norm. Classify the grounding risk of each emitted weakness:

- **`high`**: severity depends on a field norm or core-result boundary, but the reviewer supplied no applicable external grounding;
- **`med`**: the reviewer named a possible standard but did not establish its applicability;
- **`low`**: severity does not depend on a field norm, or the applicable norm is externally grounded.

The classifier evaluates whether grounding was supplied, not whether its own model knowledge says the norm is correct. It **MUST NOT** guess norm-correctness from model memory. Measurement labels must be adjudicated against the separately maintained `evals/gold/field_norm_severity` asset; the pointer is an anti-circularity boundary, not permission to expose gold material to a review run. Full tier reports counts and target-set-local shares; directional tier reports raw counts only. This is separate from FNR/FPR and does not create a quality score.

### Phase 4: Outputs

The full-tier report uses this structure:

```text
# Empirical Target Profile for <Reviewer Instance>
calibration_status: PROFILE_MEASURED
application_status: NOT_WIRED_TO_LIVE_REVIEW
profile_id: <id>
target_match: <domain; article type; venue criteria/version; rubric version;
               review mode; execution_topology_sha256>
calibration_panel_provenance: <ordered normalized_manifest_sha256 values for
                               every replay-valid measurement panel>
Gold set: n=<N>
Runs per paper: <3|5>
Cross-model: <yes/no; configuration and fallback disclosure>

## Decision-error metrics
- Balanced accuracy: <estimate and interval>
- FNR (over-harsh): <estimate and interval>
- FPR (too lenient): <estimate and interval>
- Exact four-label agreement: <count>/<N>

## Per-dimension categorical agreement
| Dimension | Agreement/confusion counts | annotated_n | missing |
| ... |

## Minor/Major boundary
<raw cells or NOT ESTIMABLE with reason>

## Severity-grounding risk
<low/med/high counts and target-set-local shares>

## Scope and transport warning
This is a measured profile on the identified target set and exact execution
topology, not universal calibration. A mismatch in any target-match field restores NOT_CALIBRATED. The current run's replay-derived
execution_topology_sha256 is one such field and must match exactly.
```

The directional report uses this structure:

```text
# Directional Calibration Readout for <Reviewer Instance>
calibration_status: NOT_CALIBRATED
Tier: directional
Gold set: n=3
Runs per paper: 1

| Paper | Gold verdict | Panel verdict | Per-seat categorical judgements | Direction |
| ... |

<raw direction counts>
<raw Minor/Major boundary cells>
<raw severity-grounding-risk counts>

Interpretation: directional observations only; no error profile, calibration,
stability, score, ranking, or transport claim is available.
```

### Phase 5: Session disclosure

A subsequent live review currently begins with this header, including after a
directional or full-tier measurement run:

```text
> calibration_status: NOT_CALIBRATED
> application_status: NOT_WIRED_TO_LIVE_REVIEW
> Live profile application is not implemented in the current release; a
> candidate profile, profile identifier, or apparent target/topology match
> cannot upgrade this live review.
> Criterion judgements are evidence-anchored but have no measured error profile.
```

The following is a **future application example only** and is not emitted by
the current Schema 6 adapter:

```text
> calibration_status: PROFILE_MEASURED
> profile_id: <id>
> profile_artifact_sha256: <raw profile digest>
> execution_topology_sha256: <profile digest, exactly equal to current replay-derived digest>
> Decision errors were measured on <N> adjudicated papers matching <target>.
> This bounded profile is not universal calibration; any target mismatch restores
> NOT_CALIBRATED. See the attached uncertainty intervals and coverage counts.
```

The current disclosure cannot be hidden or upgraded. Full-tier measurement may
produce a candidate profile, but a candidate cannot be applied to a live review
until the closed profile schema and exact-match replay validator exist. A later
directional result never replaces a full-tier candidate; neither may be reused
outside its target identity.

## Ensembling and interpretation notes

- Full-tier repeats require within-panel context separation and use majority voting for the final categorical verdict. Cross-replicate context freshness is not mechanically verified, so reports label that limitation and never infer independent repeated error processes. Per-dimension categorical agreement is reported as counts, not averaged labels.
- Directional tier is an explicit one-run exception and cannot report stability.
- Same-family evaluation can understate error. Cross-model evaluation provides stronger evidence when consented and available, but does not prove evaluator independence or correctness.
- External studies can motivate hypotheses about leniency or harshness, but their numeric gaps must not be imported as correction factors, thresholds, or target-profile measurements.
- Neither tier predicts performance outside its target identity, detects every within-paper framing failure, or replaces human editorial judgement.

## Integration with existing modes

| Mode | Interaction |
|---|---|
| `full` | Ordinary full review remains criterion-bound and defaults to `NOT_CALIBRATED`; calibration runs a separate measurement workflow. |
| `re-review` | Emit `NOT_CALIBRATED`; re-review decision rules remain unchanged. |
| `quick` | Emit `NOT_CALIBRATED` and state that reduced scope has no measured live error profile. |
| `methodology-focus` | Emit `NOT_CALIBRATED`; candidate profile application is not wired. |
| `guided` | No calibration profile is inferred from dialogue. |

## References

- Lu, C. et al. (2026). Towards end-to-end automation of AI research. *Nature* 651, 914–919. Decision-level validation motivates reporting explicit class conventions and error profiles; its numeric results are not ARS thresholds.
- Ren et al. (2026). Evaluator-independence and verifiable-subset guidance. Target-set comparison is bounded evidence, not universal calibration.
- `shared/cross_model_verification.md` — calibration transport and consent rules.
- `quality_rubrics.md` — criterion-bound judgement contract.
add-product-e2e-evalpaperclipai/paperclipAdd or extend a Paperclip full-stack runner E2E workflow, fixture, matcher, or report evidence path for local or Daytona execution.Stars: 115kUpdated Claude CodeCodexVerifiedInstall
add-runner-evalpaperclipai/paperclipAdd or extend a Paperclip Runner protocol evaluation definition, roster, assertion, or report fixture with provenance and narrow validation.Stars: 115kUpdated Claude CodeCodexVerifiedInstall
diagnose-why-work-stoppedpaperclipai/paperclipCompatibility entry point for failed, stalled, looping, or over-recovered Paperclip work. Use fix-failed-paperclip-run for the full diagnosis, regression, repair, real verification, and green-PR workflow; honor an explicitly requested diagnosis-only scope.Stars: 115kUpdated Claude CodeCodexVerifiedInstall
fix-failed-paperclip-runpaperclipai/paperclipDiagnose and fix failed, stuck, slow, or incorrectly blocked Paperclip tasks and runs. Use with a task/run URL or failure report to trace the root cause, prove a regression red then green, check second-order effects, verify the real workflow, and deliver a PR with passing checks.Stars: 115kUpdated Claude CodeCodexVerifiedInstall