Skip to content

zrwangems/dsh-zrwang-plugin / ars/skills/academic-paper-reviewer

academic-paper-reviewer

Flagged

模拟同行评审:期刊匹配审稿人 + 3 位动态领域审稿人 + 魔鬼代言人,输出编辑决定(接收/小修/大修/拒稿)与返修路线图,可做方法论专项评审与再审核验。Triggers: review paper, peer review, manuscript review, check methodology, re-review | 审稿, 同行评审, 评审论文, 检查方法, 复审。只做评审,不改写稿件。

FileKindSize
SKILL.mdskill md48.9 KB
_shared/ARCHITECTURE.mdmarkdown51.7 KB
_shared/MODE_REGISTRY.mdmarkdown6.7 KB
_shared/agents/compliance_agent.mdmarkdown9.4 KB
_shared/artifact_reproducibility_pattern.mdmarkdown9.8 KB
_shared/benchmark_report.schema.jsonconfig2.7 KB
_shared/benchmark_report_pattern.mdmarkdown8.8 KB
_shared/bibliographic_integrity_signals.mdmarkdown8.1 KB
_shared/collaboration_depth_rubric.mdmarkdown11.5 KB
_shared/compliance_checkpoint_protocol.mdmarkdown8.9 KB
_shared/compliance_report.schema.jsonconfig8.1 KB
_shared/contracts/README.mdmarkdown54.7 KB
_shared/contracts/activity/adjudication_activity_input.schema.jsonconfig18.5 KB
_shared/contracts/activity/adjudication_activity_store.schema.jsonconfig18.7 KB
_shared/contracts/audit/audit_jsonl.schema.jsonconfig5.4 KB
_shared/contracts/audit/audit_sidecar.schema.jsonconfig8.4 KB
_shared/contracts/audit/audit_verdict.schema.jsonconfig5.4 KB
_shared/contracts/audit/cross_document_consistency_advisory.schema.jsonconfig20.4 KB
_shared/contracts/audit/cross_document_consistency_advisory_draft.schema.jsonconfig23.4 KB
_shared/contracts/audit/cross_document_source_manifest.schema.jsonconfig6.3 KB
_shared/contracts/audit/tortured_phrase_advisory.schema.jsonconfig30.9 KB
_shared/contracts/audit/tortured_phrase_snapshot.schema.jsonconfig4.9 KB
_shared/contracts/audit/tortured_phrase_snapshot_manifest.schema.jsonconfig7.3 KB
_shared/contracts/capability/stage_capability_matrix.jsonconfig23.5 KB
_shared/contracts/claim_standing/candidate_ledger.schema.jsonconfig23.1 KB
_shared/contracts/claim_standing/query_plan.schema.jsonconfig12.7 KB
_shared/contracts/claim_standing/query_plan_v1_1.schema.jsonconfig20.3 KB
_shared/contracts/claim_standing/retrieval_input.schema.jsonconfig14.4 KB
_shared/contracts/claim_standing/stance_record.schema.jsonconfig11.7 KB
_shared/contracts/claim_standing/transmission_ledger.schema.jsonconfig7.6 KB
_shared/contracts/cross_model/codex_citation_receipt.schema.jsonconfig4.7 KB
_shared/contracts/cross_model/codex_citation_request.schema.jsonconfig986 B
_shared/contracts/cross_model/promotion_bakeoff_sealed_commitment.schema.jsonconfig1.5 KB
_shared/contracts/cross_model/promotion_bakeoff_sealed_reveal.schema.jsonconfig1.5 KB
_shared/contracts/degradation_registry.jsonconfig42.5 KB
_shared/contracts/evaluator/full.jsonconfig6.3 KB
_shared/contracts/evidence/claim_registry.schema.jsonconfig2.3 KB
_shared/contracts/evidence/claim_registry_coverage_report.schema.jsonconfig4.8 KB
_shared/contracts/evidence/evidence_row.schema.jsonconfig14.3 KB
_shared/contracts/evidence/evidence_row_v1_1.schema.jsonconfig11.4 KB
_shared/contracts/evidence/evidence_row_v1_2.schema.jsonconfig21.8 KB
_shared/contracts/evidence/evidence_row_v1_3.schema.jsonconfig10.7 KB
_shared/contracts/human_subjects/authority_profile_registry.schema.jsonconfig16.7 KB
_shared/contracts/human_subjects/committee_correspondence.schema.jsonconfig9.9 KB
_shared/contracts/human_subjects/content_coverage_advisory.schema.jsonconfig19.9 KB
_shared/contracts/human_subjects/irb_context_record.schema.jsonconfig12.1 KB
_shared/contracts/human_subjects/resolved_authority_context.schema.jsonconfig11.8 KB
_shared/contracts/human_subjects/review_pathway_rule_trace.schema.jsonconfig9.5 KB
_shared/contracts/human_subjects/review_pathway_trace_request.schema.jsonconfig4.8 KB
_shared/contracts/human_subjects/submission_packet_inventory.schema.jsonconfig8.5 KB
_shared/contracts/human_subjects/submission_packet_manifest.schema.jsonconfig32.2 KB
_shared/contracts/passport/audit_artifact_entry.schema.jsonconfig10.8 KB
_shared/contracts/passport/bibliographic_integrity_signal.schema.jsonconfig48.9 KB
_shared/contracts/passport/citation_provenance.schema.jsonconfig4.8 KB
_shared/contracts/passport/citation_verification_summary.schema.jsonconfig8.2 KB
_shared/contracts/passport/claim_audit_result.schema.jsonconfig7.8 KB
_shared/contracts/passport/claim_drift.schema.jsonconfig3.8 KB
_shared/contracts/passport/claim_intent_manifest.schema.jsonconfig6.6 KB
_shared/contracts/passport/constraint_violation.schema.jsonconfig3.6 KB
_shared/contracts/passport/excluded_source_entry.schema.jsonconfig1.7 KB
_shared/contracts/passport/experiment_alignment_result.schema.jsonconfig4.8 KB
_shared/contracts/passport/experiment_provenance_entry.schema.jsonconfig10.0 KB
_shared/contracts/passport/human_read_log.schema.jsonconfig3.8 KB
_shared/contracts/passport/inquiry_ledger_ref.schema.jsonconfig881 B
_shared/contracts/passport/literature_corpus_entry.schema.jsonconfig30.3 KB
_shared/contracts/passport/preregistration_artifact.schema.jsonconfig4.3 KB
_shared/contracts/passport/rejection_log.schema.jsonconfig3.3 KB
_shared/contracts/passport/reset_ledger_entry.schema.jsonconfig7.1 KB
_shared/contracts/passport/run_ledger.schema.jsonconfig8.0 KB
_shared/contracts/passport/standing_constraint_entry.schema.jsonconfig2.6 KB
_shared/contracts/passport/temporal_audit_results.schema.jsonconfig8.8 KB
_shared/contracts/passport/terminal_policies.schema.jsonconfig8.1 KB
_shared/contracts/passport/timeline.schema.jsonconfig4.7 KB
_shared/contracts/passport/uncited_assertion.schema.jsonconfig3.3 KB
_shared/contracts/passport/uncited_audit_failure.schema.jsonconfig4.4 KB
_shared/contracts/passport/user_attested_read_resolution.schema.jsonconfig3.2 KB
_shared/contracts/passport/version_records.schema.jsonconfig4.9 KB
_shared/contracts/patch/block_manifest.schema.jsonconfig2.1 KB
_shared/contracts/patch/legacy/v1_0/revision_patch.schema.jsonconfig5.3 KB
_shared/contracts/patch/revision_patch.schema.jsonconfig8.9 KB
_shared/contracts/pdf/pdf_content_classifier_diagnostic.schema.jsonconfig1.2 KB
_shared/contracts/pdf/pdf_content_classifier_worker.schema.jsonconfig2.2 KB
_shared/contracts/pdf/pdf_read_preflight.schema.jsonconfig6.8 KB
_shared/contracts/re_review/input_manifest.schema.jsonconfig7.6 KB
_shared/contracts/re_review/legacy/v1_0/input_manifest.schema.jsonconfig6.2 KB
_shared/contracts/re_review/legacy/v1_0/precommitment.schema.jsonconfig8.9 KB
_shared/contracts/re_review/legacy/v1_0/traceability.schema.jsonconfig34.2 KB
_shared/contracts/re_review/legacy/v1_0/verdict_record.schema.jsonconfig10.5 KB
_shared/contracts/re_review/precommitment.schema.jsonconfig8.6 KB
_shared/contracts/re_review/traceability.schema.jsonconfig37.7 KB
_shared/contracts/re_review/verdict_record.schema.jsonconfig10.5 KB
_shared/contracts/research_workflow/inquiry_branch_ledger.schema.jsonconfig17.0 KB
_shared/contracts/research_workflow/research_workflow_profile.schema.jsonconfig7.2 KB
_shared/contracts/research_workflow/research_workflow_profile_selection_receipt.schema.jsonconfig3.9 KB
_shared/contracts/review_target/constructive_review_findings.schema.jsonconfig7.0 KB
_shared/contracts/review_target/criteria_registry.schema.jsonconfig4.0 KB
_shared/contracts/review_target/review_criteria_binding_manifest.schema.jsonconfig7.9 KB
_shared/contracts/review_target/review_criteria_source_receipt.schema.jsonconfig5.6 KB
_shared/contracts/review_target/review_target_context.schema.jsonconfig4.5 KB
_shared/contracts/review_target/review_target_declaration.schema.jsonconfig4.6 KB
_shared/contracts/reviewer/full.jsonconfig3.7 KB
_shared/contracts/reviewer/methodology_focus.jsonconfig2.2 KB
_shared/contracts/reviewer/review_panel_provenance.schema.jsonconfig7.8 KB
_shared/contracts/reviewer/review_panel_provenance_carrier.schema.jsonconfig4.1 KB
_shared/contracts/reviewer/review_panel_provenance_input.schema.jsonconfig3.5 KB
_shared/contracts/revision/author_adjudication.schema.jsonconfig7.8 KB
_shared/contracts/revision/author_adjudication_input.schema.jsonconfig1.2 KB
_shared/contracts/revision/claim_strength_drift_disposition.schema.jsonconfig3.6 KB
_shared/contracts/revision/claim_strength_drift_disposition_input.schema.jsonconfig3.1 KB
_shared/contracts/revision/claim_strength_drift_findings.schema.jsonconfig4.2 KB
_shared/contracts/revision/claim_surface_manifest.schema.jsonconfig3.3 KB
_shared/contracts/revision/integrity_correction_authorization.schema.jsonconfig4.3 KB
_shared/contracts/revision/integrity_correction_authorization_input.schema.jsonconfig1.5 KB
_shared/contracts/revision/integrity_correction_list.schema.jsonconfig1.8 KB
_shared/contracts/revision/integrity_pass_receipt.schema.jsonconfig809 B
_shared/contracts/revision/revision_evidence_bundle.schema.jsonconfig4.9 KB
_shared/contracts/revision/revision_roadmap.schema.jsonconfig11.2 KB
_shared/contracts/submission/format_profile.example.yamlconfig971 B
_shared/contracts/submission/format_profile.schema.jsonconfig4.6 KB
_shared/contracts/submission/submission_verification_report.schema.jsonconfig10.3 KB
_shared/contracts/submission/venue_profile.schema.jsonconfig4.1 KB
_shared/contracts/writer/full.jsonconfig3.6 KB
_shared/cross_model_verification.mdmarkdown83.3 KB
_shared/description-original.mdmarkdown1.0 KB
_shared/dsh-runtime.mdmarkdown1.5 KB
_shared/evals_lift_report.schema.jsonconfig5.0 KB
_shared/ground_truth_isolation_pattern.mdmarkdown16.0 KB
_shared/handoff_schemas.mdmarkdown96.9 KB
_shared/human_subjects_authority_registry.jsonconfig49.5 KB
_shared/mode_spectrum.mdmarkdown5.2 KB
_shared/model_tiering.mdmarkdown12.6 KB
_shared/output_language_pair.mdmarkdown7.7 KB
_shared/policy_data/nature_policy.mdmarkdown3.9 KB
_shared/prisma_trAIce_protocol.mdmarkdown10.2 KB
_shared/raise_framework.mdmarkdown7.5 KB
_shared/references/authority_content_coverage_advisory_protocol.mdmarkdown14.0 KB
_shared/references/claim_standing_candidate_ledger_protocol.mdmarkdown19.2 KB
_shared/references/claim_strength_ladder.mdmarkdown5.5 KB
_shared/references/cross_document_consistency_advisory_protocol.mdmarkdown13.5 KB
_shared/references/evidence_row_protocol.mdmarkdown14.4 KB
_shared/references/firm_rules.mdmarkdown11.0 KB
_shared/references/human_subjects_authority_protocol.mdmarkdown13.1 KB
_shared/references/intent_clarification_protocol.mdmarkdown11.9 KB
_shared/references/irb_terminology_glossary.mdmarkdown12.4 KB
_shared/references/per_source_method_weaknesses.mdmarkdown4.2 KB
_shared/references/protected_hedging_phrases.mdmarkdown8.9 KB
_shared/references/psychometric_terminology_glossary.mdmarkdown8.7 KB
_shared/references/review_criteria_consumer_protocol.mdmarkdown10.3 KB
_shared/references/review_form_note.mdmarkdown11.3 KB
_shared/references/review_pathway_rule_trace_protocol.mdmarkdown8.3 KB
_shared/references/routing_core.mdmarkdown5.7 KB
_shared/references/submission_packet_manifest_protocol.mdmarkdown13.6 KB
_shared/references/word_count_conventions.mdmarkdown9.0 KB
_shared/research_workflow_profiles/field_general.jsonconfig1.2 KB
_shared/review_criteria_registry.jsonconfig11.5 KB
_shared/review_criteria_sources/msr-2027-technical-papers.2026-08-24.jsonconfig2.0 KB
_shared/review_criteria_sources/sigsoft-empirical-standards.2026-08-24.jsonconfig1.4 KB
_shared/sprint_contract.schema.jsonconfig19.0 KB
_shared/style_calibration_protocol.mdmarkdown6.9 KB
_shared/templates/codex_audit_multifile_template.mdmarkdown16.7 KB
agents/devils_advocate_reviewer_agent.mdmarkdown42.6 KB
agents/domain_reviewer_agent.mdmarkdown34.0 KB
agents/editorial_synthesizer_agent.mdmarkdown42.4 KB
agents/eic_agent.mdmarkdown28.8 KB
agents/field_analyst_agent.mdmarkdown11.9 KB
agents/methodology_reviewer_agent.mdmarkdown44.6 KB
agents/perspective_reviewer_agent.mdmarkdown34.3 KB
examples/hei_paper_review_example.mdmarkdown33.8 KB
examples/interdisciplinary_review_example.mdmarkdown27.6 KB
examples/subclaim_decomposition_example.mdmarkdown8.3 KB
references/calibration_mode_protocol.mdmarkdown16.1 KB
references/changelog.mdmarkdown2.0 KB
references/editorial_decision_standards.mdmarkdown13.5 KB
references/guided_mode_protocol.mdmarkdown2.6 KB
references/integration_guide.mdmarkdown1.3 KB
references/quality_rubrics.mdmarkdown6.9 KB
references/re_review_mode_protocol.mdmarkdown52.6 KB
references/review_criteria_framework.mdmarkdown6.8 KB
references/review_panel_provenance_protocol.mdmarkdown10.1 KB
references/review_quality_thinking.mdmarkdown3.9 KB
references/reviewer_sprint_prompt_source.mdmarkdown48.5 KB
references/sprint_contract_protocol.mdmarkdown32.7 KB
references/statistical_reporting_standards.mdmarkdown30.4 KB
references/top_journals_by_field.mdmarkdown13.2 KB
scripts/_claim_audit_constants.pyscript13.7 KB
scripts/_markdown_lint_util.pyscript9.0 KB
scripts/_next_verified_at_ms.pyscript6.5 KB
scripts/_skill_lint.pyscript13.6 KB
scripts/_text_similarity.pyscript16.0 KB
scripts/ars_cache_invalidate.pyscript1.3 KB
scripts/ars_mark_read.pyscript18.2 KB
scripts/ars_write_scope_guard.pyscript30.9 KB
scripts/arxiv_client.pyscript9.4 KB
scripts/audit_panel_claims.pyscript17.3 KB
scripts/audit_snapshot.pyscript23.5 KB
scripts/bibliographic_integrity_signals.pyscript32.2 KB
scripts/build_claim_standing_candidate_ledger.pyscript54.3 KB
scripts/build_claim_standing_query_plan.pyscript25.0 KB
scripts/build_content_coverage_advisory.pyscript45.1 KB
scripts/build_cross_document_consistency_advisory.pyscript77.9 KB

references/calibration_mode_protocol.md

View on GitHub
> 输出语言(本 DSH 集成强制):所有面向用户的文字、稿件、摘要、评审与报告一律使用**简体中文**;英文术语与引文保持英文。文中的 `zh-TW` / `zh-tw-en` 只是上游语言代码与脚本参数,不代表要用繁体输出。用户明确要求其他语言时以用户为准。
>
> 多角色执行(评审团、十阶段流水线、角色扮演)必须使用 DSH 的 Agent Team 工具(`spawn_teammate`、`team_task_*`、`wait_agent`、`send_message`);当前会话看不到这些工具,说明用户没开 Agent Team,必须先提示用户开启,不得退化成单会话顺序扮演。
# Calibration Mode Protocol

**Parent skill**: `academic-paper-reviewer`
**Mode name**: `calibration`
**Purpose**: Measure a bounded decision-error profile against a user-adjudicated target set, or obtain a cheaper directional readout. This protocol never turns criterion judgements into an absolute quality score.

## Epistemic status

An ordinary review has `calibration_status: NOT_CALIBRATED`. Its categorical judgements are tied to named criteria and evidence in one manuscript; they do not establish a stable interval scale, paper ranking, acceptance probability, or cross-session reproducibility.

The full tier may produce a candidate empirical target profile only after completing the protocol against the current domain, article type, venue criteria, rubric version, review mode, and exact replay-derived `execution_topology_sha256`. `PROFILE_MEASURED` means only that errors were measured on that bounded target set. It is not a claim of universal calibration and must not be transported to a materially different target or review topology.

**Current application boundary:** candidate profile production and live-review
profile application are separate operations. Until a closed, hash-bound target
profile schema plus replay validator is shipped, reviewer seats and the current
Schema 6 package adapter emit `calibration_status: NOT_CALIBRATED`; a prose
profile ID or apparent topology match cannot upgrade them. The full-tier report
may describe its measured candidate profile, but must label it
`application_status: NOT_WIRED_TO_LIVE_REVIEW`.

The directional tier always reports `calibration_status: NOT_CALIBRATED`. Three single panels provide directional observations, not an error profile.

## Inputs

1. **Tier**: `full` by default. Use `directional` only when explicitly selected.
2. **Empirical target set**:
   - Full: 5–20 user-adjudicated papers, preferably 10–15, with at least one acceptable-side and one reject-side label.
   - Directional: exactly three papers—one `minor_revision`, one `major_revision`, and one extreme anchor (`accept` or `reject`).
3. **Per-paper fields**:
   - manuscript path or text;
   - adjudicated verdict: `accept`, `minor_revision`, `major_revision`, `reject`, or legacy `borderline` in the full tier only;
   - venue, domain, and article-type context;
   - optional `per_dimension_gold_judgements`, using the same criterion-bound categories as `quality_rubrics.md`, with the criterion source and adjudication rationale.
4. **Target-profile identity**: domain, article type, venue criteria/version, rubric version, review mode, target-set identifier, adjudication date, and one exact `execution_topology_sha256` derived from actual completed panel provenance. A prose configuration label or intended route is not a match key.
5. **Session persistence**: session-only. Do not cache manuscripts or profiles across sessions.

### Gold-label isolation

Gold verdicts, dimension judgements, rationales, and human assessments must not enter field-analyst, reviewer, or synthesizer context. Join them only after a panel verdict is frozen. The substrate plan must also be fixed without consulting gold material.

This isolation applies to transport as well as prompt construction: no gold
verdict, dimension judgement, rationale, human assessment, or target-set
outcome may enter a provider payload or determine the substrate plan. The join
is post-freeze only, and joined material cannot be fed back into another panel
within the same attempt.

## Process

### Phase 0: Intake

- Full tier: verify 5–20 papers and both sides of the binary decision boundary. Warn that small or clustered target sets yield uncertain, non-transportable estimates. Legacy `borderline` papers are reported separately and excluded from binary metrics.
- Directional tier: verify exactly the required three-paper composition, one run per paper, and no `borderline` item. State that the result remains `NOT_CALIBRATED`.
- Never infer the tier from paper count; ask when the user's selection is missing.

### Phase 1: Run panels

**Full tier.** Run 3 or 5 panel replicates per paper (default 5), then derive the panel verdict by majority vote. Within each replicate, the five seats must have distinct recorded invocation-context IDs. The current provenance artifact checks only that within-panel separation; it does not compare IDs against earlier replicates and therefore does not establish cross-replicate freshness or independent error processes. Preserve each seat's criterion-bound judgements and evidence; do not convert them to numbers or average them. Report exact-agreement and disagreement patterns for a dimension only when the matching gold judgement exists.

**Directional tier.** Run one panel per paper with distinct recorded invocation-context IDs among its five seats. Preserve the exact panel verdict and each scoring seat's categorical dimension judgements as emitted. Do not ensemble, manufacture variance, combine labels into a panel score, or rank the papers.

**Cross-model verification and actual provenance.** `ARS_CROSS_MODEL` is
default-on for calibration mode. Follow
`_shared/cross_model_verification.md` § Calibration transport exception. Before
any provider call, run its closed calibration data-fence collision preflight
independently on the raw reviewer-configuration bytes and raw manuscript bytes.
A collision refuses the entire attempt before transport: do not send either
payload and do not escape, strip, rewrite, truncate, switch delimiters, or
silently fall back.

Create an `attempt_id`, lock one `substrate_plan` before any gold material is
consulted, and use the same transport and substrate for every paper and
replicate. For every completed panel, build and replay-validate
`review-panel-provenance/1.0` from actual seat executions. All result-producing
panels in one calibration attempt MUST have one identical
`execution_topology_sha256`; every artifact must derive `fresh_context: true`
under its fixed `within_panel_attempt_only` scope. The builder receives no
attempt-history ledger: even if invocation-context IDs differ within each
artifact, it cannot detect reuse across papers or replicates. Every candidate
profile and readout must disclose that cross-replicate freshness is unverified
and must not describe the repeated panels as independent. If consent, configuration, or
non-content transport preflight is unavailable before the attempt begins, lock
all seats to the primary family and execute the complete schedule on that
homogeneous plan.

A mid-attempt dispatch or substrate failure, topology mismatch, unknown
required provenance observation, within-panel context reuse, or invalid provenance
artifact invalidates the whole attempt. Every completed panel becomes
diagnostic-only and must not enter any aggregate. The only result-producing
retry uses a new `attempt_id`, an empty aggregate, and restarts at paper 1 /
replicate 1 on one homogeneous plan. Never resume the failed attempt, mix
transports or substrates, or emit a profile or directional readout from an
incomplete, mixed-substrate, mixed-topology, or provenance-unresolved attempt.

### Phase 2: Full-tier decision-error profile

Map adjudicated and panel verdicts as follows: Accept/Minor Revision are the acceptable side; Major Revision/Reject are the reject side. With positive meaning acceptable:

| Metric | Meaning | Reporting boundary |
|---|---|---|
| Balanced accuracy | Mean of sensitivity on the two sides | Point estimate plus bootstrap interval |
| FNR | Acceptable-side papers judged Major/Reject (over-harsh) | Point estimate plus bootstrap interval |
| FPR | Reject-side papers judged Accept/Minor (too lenient) | Point estimate plus bootstrap interval |
| Exact four-label agreement | Exact verdict matches | Count and share, with target-set size |

Do not report AUC: there is no continuous rubric score. Do not infer an ordinal paper ranking from the four editorial labels.

For each dimension with adjudicated gold judgements, report a categorical agreement table and `annotated_n=<n>/<N>, missing=<N-n>`. With no annotated gold for a dimension, report `NOT COMPUTABLE`. Do not assign distances between categories, average the labels, or claim a gold-set-wide dimension result from a subset.

### Phase 2.5: Minor/Major boundary (both tiers)

When both `minor_revision` and `major_revision` gold examples exist, report raw counts:

| Gold \ predicted side | Accept + Minor | Major + Reject |
|---|---:|---:|
| Minor Revision | stayed minor-side | harsh crossing |
| Major Revision | lenient crossing | stayed major-side |

If either side is absent in a full-tier target set, report `NOT ESTIMABLE — target set lacks both sides of the Minor/Major boundary`. Never render an all-zero table as evidence of no confusion.

### Phase 2.6: Directional reporting boundary

The directional readout may report only:

- each paper's exact gold verdict, exact panel verdict, per-seat categorical criterion judgements, and `lenient` / `exact` / `harsh` direction;
- raw direction counts;
- the raw Minor/Major boundary cells; and
- raw low/med/high severity-grounding-risk counts from Phase 3.5.

It must not report balanced accuracy, FNR, FPR, AUC, confidence intervals, stability, per-dimension error rates, numeric baseline comparisons, or any measured-profile claim.

### Phase 3.5: Severity-miscalibration measurement (#215)

Decision-level error does not capture findings whose severity relies on an asserted field norm. Classify the grounding risk of each emitted weakness:

- **`high`**: severity depends on a field norm or core-result boundary, but the reviewer supplied no applicable external grounding;
- **`med`**: the reviewer named a possible standard but did not establish its applicability;
- **`low`**: severity does not depend on a field norm, or the applicable norm is externally grounded.

The classifier evaluates whether grounding was supplied, not whether its own model knowledge says the norm is correct. It **MUST NOT** guess norm-correctness from model memory. Measurement labels must be adjudicated against the separately maintained `evals/gold/field_norm_severity` asset; the pointer is an anti-circularity boundary, not permission to expose gold material to a review run. Full tier reports counts and target-set-local shares; directional tier reports raw counts only. This is separate from FNR/FPR and does not create a quality score.

### Phase 4: Outputs

The full-tier report uses this structure:

```text
# Empirical Target Profile for <Reviewer Instance>
calibration_status: PROFILE_MEASURED
application_status: NOT_WIRED_TO_LIVE_REVIEW
profile_id: <id>
target_match: <domain; article type; venue criteria/version; rubric version;
               review mode; execution_topology_sha256>
calibration_panel_provenance: <ordered normalized_manifest_sha256 values for
                               every replay-valid measurement panel>
Gold set: n=<N>
Runs per paper: <3|5>
Cross-model: <yes/no; configuration and fallback disclosure>

## Decision-error metrics
- Balanced accuracy: <estimate and interval>
- FNR (over-harsh): <estimate and interval>
- FPR (too lenient): <estimate and interval>
- Exact four-label agreement: <count>/<N>

## Per-dimension categorical agreement
| Dimension | Agreement/confusion counts | annotated_n | missing |
| ... |

## Minor/Major boundary
<raw cells or NOT ESTIMABLE with reason>

## Severity-grounding risk
<low/med/high counts and target-set-local shares>

## Scope and transport warning
This is a measured profile on the identified target set and exact execution
topology, not universal calibration. A mismatch in any target-match field restores NOT_CALIBRATED. The current run's replay-derived
execution_topology_sha256 is one such field and must match exactly.
```

The directional report uses this structure:

```text
# Directional Calibration Readout for <Reviewer Instance>
calibration_status: NOT_CALIBRATED
Tier: directional
Gold set: n=3
Runs per paper: 1

| Paper | Gold verdict | Panel verdict | Per-seat categorical judgements | Direction |
| ... |

<raw direction counts>
<raw Minor/Major boundary cells>
<raw severity-grounding-risk counts>

Interpretation: directional observations only; no error profile, calibration,
stability, score, ranking, or transport claim is available.
```

### Phase 5: Session disclosure

A subsequent live review currently begins with this header, including after a
directional or full-tier measurement run:

```text
> calibration_status: NOT_CALIBRATED
> application_status: NOT_WIRED_TO_LIVE_REVIEW
> Live profile application is not implemented in the current release; a
> candidate profile, profile identifier, or apparent target/topology match
> cannot upgrade this live review.
> Criterion judgements are evidence-anchored but have no measured error profile.
```

The following is a **future application example only** and is not emitted by
the current Schema 6 adapter:

```text
> calibration_status: PROFILE_MEASURED
> profile_id: <id>
> profile_artifact_sha256: <raw profile digest>
> execution_topology_sha256: <profile digest, exactly equal to current replay-derived digest>
> Decision errors were measured on <N> adjudicated papers matching <target>.
> This bounded profile is not universal calibration; any target mismatch restores
> NOT_CALIBRATED. See the attached uncertainty intervals and coverage counts.
```

The current disclosure cannot be hidden or upgraded. Full-tier measurement may
produce a candidate profile, but a candidate cannot be applied to a live review
until the closed profile schema and exact-match replay validator exist. A later
directional result never replaces a full-tier candidate; neither may be reused
outside its target identity.

## Ensembling and interpretation notes

- Full-tier repeats require within-panel context separation and use majority voting for the final categorical verdict. Cross-replicate context freshness is not mechanically verified, so reports label that limitation and never infer independent repeated error processes. Per-dimension categorical agreement is reported as counts, not averaged labels.
- Directional tier is an explicit one-run exception and cannot report stability.
- Same-family evaluation can understate error. Cross-model evaluation provides stronger evidence when consented and available, but does not prove evaluator independence or correctness.
- External studies can motivate hypotheses about leniency or harshness, but their numeric gaps must not be imported as correction factors, thresholds, or target-profile measurements.
- Neither tier predicts performance outside its target identity, detects every within-paper framing failure, or replaces human editorial judgement.

## Integration with existing modes

| Mode | Interaction |
|---|---|
| `full` | Ordinary full review remains criterion-bound and defaults to `NOT_CALIBRATED`; calibration runs a separate measurement workflow. |
| `re-review` | Emit `NOT_CALIBRATED`; re-review decision rules remain unchanged. |
| `quick` | Emit `NOT_CALIBRATED` and state that reduced scope has no measured live error profile. |
| `methodology-focus` | Emit `NOT_CALIBRATED`; candidate profile application is not wired. |
| `guided` | No calibration profile is inferred from dialogue. |

## References

- Lu, C. et al. (2026). Towards end-to-end automation of AI research. *Nature* 651, 914–919. Decision-level validation motivates reporting explicit class conventions and error profiles; its numeric results are not ARS thresholds.
- Ren et al. (2026). Evaluator-independence and verifiable-subset guidance. Target-set comparison is bounded evidence, not universal calibration.
- `_shared/cross_model_verification.md` — calibration transport and consent rules.
- `quality_rubrics.md` — criterion-bound judgement contract.
add-product-e2e-evalpaperclipai/paperclipAdd or extend a Paperclip full-stack runner E2E workflow, fixture, matcher, or report evidence path for local or Daytona execution.Stars: 115kUpdated Claude CodeCodexVerifiedInstall
add-runner-evalpaperclipai/paperclipAdd or extend a Paperclip Runner protocol evaluation definition, roster, assertion, or report fixture with provenance and narrow validation.Stars: 115kUpdated Claude CodeCodexVerifiedInstall
diagnose-why-work-stoppedpaperclipai/paperclipCompatibility entry point for failed, stalled, looping, or over-recovered Paperclip work. Use fix-failed-paperclip-run for the full diagnosis, regression, repair, real verification, and green-PR workflow; honor an explicitly requested diagnosis-only scope.Stars: 115kUpdated Claude CodeCodexVerifiedInstall
fix-failed-paperclip-runpaperclipai/paperclipDiagnose and fix failed, stuck, slow, or incorrectly blocked Paperclip tasks and runs. Use with a task/run URL or failure report to trace the root cause, prove a regression red then green, check second-order effects, verify the real workflow, and deliver a PR with passing checks.Stars: 115kUpdated Claude CodeCodexVerifiedInstall