Skip to content

zrwangems/dsh-zrwang-plugin / ars/skills/sr-screener

sr-screener

Flagged

系统综述文献筛选:把方案或 PROSPERO 协议转成资格规则,双盲筛选标题摘要与全文,第三人裁决冲突,输出 PRISMA 2020 计数、EndNote/Zotero RIS 分组与可交付给论文技能的 literature_corpus。Triggers: systematic review screening, screen papers, PRISMA counts, pilot screening, adjudicate conflicts | 文献筛选, 系统综述筛选, PRISMA 计数, 筛选冲突裁决。

FileKindSize
SKILL.mdskill md33.1 KB
_shared/ARCHITECTURE.mdmarkdown51.7 KB
_shared/MODE_REGISTRY.mdmarkdown6.7 KB
_shared/agents/compliance_agent.mdmarkdown9.4 KB
_shared/artifact_reproducibility_pattern.mdmarkdown9.8 KB
_shared/benchmark_report.schema.jsonconfig2.7 KB
_shared/benchmark_report_pattern.mdmarkdown8.8 KB
_shared/bibliographic_integrity_signals.mdmarkdown8.1 KB
_shared/collaboration_depth_rubric.mdmarkdown11.5 KB
_shared/compliance_checkpoint_protocol.mdmarkdown8.9 KB
_shared/compliance_report.schema.jsonconfig8.1 KB
_shared/contracts/README.mdmarkdown54.7 KB
_shared/contracts/activity/adjudication_activity_input.schema.jsonconfig18.5 KB
_shared/contracts/activity/adjudication_activity_store.schema.jsonconfig18.7 KB
_shared/contracts/audit/audit_jsonl.schema.jsonconfig5.4 KB
_shared/contracts/audit/audit_sidecar.schema.jsonconfig8.4 KB
_shared/contracts/audit/audit_verdict.schema.jsonconfig5.4 KB
_shared/contracts/audit/cross_document_consistency_advisory.schema.jsonconfig20.4 KB
_shared/contracts/audit/cross_document_consistency_advisory_draft.schema.jsonconfig23.4 KB
_shared/contracts/audit/cross_document_source_manifest.schema.jsonconfig6.3 KB
_shared/contracts/audit/tortured_phrase_advisory.schema.jsonconfig30.9 KB
_shared/contracts/audit/tortured_phrase_snapshot.schema.jsonconfig4.9 KB
_shared/contracts/audit/tortured_phrase_snapshot_manifest.schema.jsonconfig7.3 KB
_shared/contracts/capability/stage_capability_matrix.jsonconfig23.5 KB
_shared/contracts/claim_standing/candidate_ledger.schema.jsonconfig23.1 KB
_shared/contracts/claim_standing/query_plan.schema.jsonconfig12.7 KB
_shared/contracts/claim_standing/query_plan_v1_1.schema.jsonconfig20.3 KB
_shared/contracts/claim_standing/retrieval_input.schema.jsonconfig14.4 KB
_shared/contracts/claim_standing/stance_record.schema.jsonconfig11.7 KB
_shared/contracts/claim_standing/transmission_ledger.schema.jsonconfig7.6 KB
_shared/contracts/cross_model/codex_citation_receipt.schema.jsonconfig4.7 KB
_shared/contracts/cross_model/codex_citation_request.schema.jsonconfig986 B
_shared/contracts/cross_model/promotion_bakeoff_sealed_commitment.schema.jsonconfig1.5 KB
_shared/contracts/cross_model/promotion_bakeoff_sealed_reveal.schema.jsonconfig1.5 KB
_shared/contracts/degradation_registry.jsonconfig42.5 KB
_shared/contracts/evaluator/full.jsonconfig6.3 KB
_shared/contracts/evidence/claim_registry.schema.jsonconfig2.3 KB
_shared/contracts/evidence/claim_registry_coverage_report.schema.jsonconfig4.8 KB
_shared/contracts/evidence/evidence_row.schema.jsonconfig14.3 KB
_shared/contracts/evidence/evidence_row_v1_1.schema.jsonconfig11.4 KB
_shared/contracts/evidence/evidence_row_v1_2.schema.jsonconfig21.8 KB
_shared/contracts/evidence/evidence_row_v1_3.schema.jsonconfig10.7 KB
_shared/contracts/human_subjects/authority_profile_registry.schema.jsonconfig16.7 KB
_shared/contracts/human_subjects/committee_correspondence.schema.jsonconfig9.9 KB
_shared/contracts/human_subjects/content_coverage_advisory.schema.jsonconfig19.9 KB
_shared/contracts/human_subjects/irb_context_record.schema.jsonconfig12.1 KB
_shared/contracts/human_subjects/resolved_authority_context.schema.jsonconfig11.8 KB
_shared/contracts/human_subjects/review_pathway_rule_trace.schema.jsonconfig9.5 KB
_shared/contracts/human_subjects/review_pathway_trace_request.schema.jsonconfig4.8 KB
_shared/contracts/human_subjects/submission_packet_inventory.schema.jsonconfig8.5 KB
_shared/contracts/human_subjects/submission_packet_manifest.schema.jsonconfig32.2 KB
_shared/contracts/passport/audit_artifact_entry.schema.jsonconfig10.8 KB
_shared/contracts/passport/bibliographic_integrity_signal.schema.jsonconfig48.9 KB
_shared/contracts/passport/citation_provenance.schema.jsonconfig4.8 KB
_shared/contracts/passport/citation_verification_summary.schema.jsonconfig8.2 KB
_shared/contracts/passport/claim_audit_result.schema.jsonconfig7.8 KB
_shared/contracts/passport/claim_drift.schema.jsonconfig3.8 KB
_shared/contracts/passport/claim_intent_manifest.schema.jsonconfig6.6 KB
_shared/contracts/passport/constraint_violation.schema.jsonconfig3.6 KB
_shared/contracts/passport/excluded_source_entry.schema.jsonconfig1.7 KB
_shared/contracts/passport/experiment_alignment_result.schema.jsonconfig4.8 KB
_shared/contracts/passport/experiment_provenance_entry.schema.jsonconfig10.0 KB
_shared/contracts/passport/human_read_log.schema.jsonconfig3.8 KB
_shared/contracts/passport/inquiry_ledger_ref.schema.jsonconfig881 B
_shared/contracts/passport/literature_corpus_entry.schema.jsonconfig30.3 KB
_shared/contracts/passport/preregistration_artifact.schema.jsonconfig4.3 KB
_shared/contracts/passport/rejection_log.schema.jsonconfig3.3 KB
_shared/contracts/passport/reset_ledger_entry.schema.jsonconfig7.1 KB
_shared/contracts/passport/run_ledger.schema.jsonconfig8.0 KB
_shared/contracts/passport/standing_constraint_entry.schema.jsonconfig2.6 KB
_shared/contracts/passport/temporal_audit_results.schema.jsonconfig8.8 KB
_shared/contracts/passport/terminal_policies.schema.jsonconfig8.1 KB
_shared/contracts/passport/timeline.schema.jsonconfig4.7 KB
_shared/contracts/passport/uncited_assertion.schema.jsonconfig3.3 KB
_shared/contracts/passport/uncited_audit_failure.schema.jsonconfig4.4 KB
_shared/contracts/passport/user_attested_read_resolution.schema.jsonconfig3.2 KB
_shared/contracts/passport/version_records.schema.jsonconfig4.9 KB
_shared/contracts/patch/block_manifest.schema.jsonconfig2.1 KB
_shared/contracts/patch/legacy/v1_0/revision_patch.schema.jsonconfig5.3 KB
_shared/contracts/patch/revision_patch.schema.jsonconfig8.9 KB
_shared/contracts/pdf/pdf_content_classifier_diagnostic.schema.jsonconfig1.2 KB
_shared/contracts/pdf/pdf_content_classifier_worker.schema.jsonconfig2.2 KB
_shared/contracts/pdf/pdf_read_preflight.schema.jsonconfig6.8 KB
_shared/contracts/re_review/input_manifest.schema.jsonconfig7.6 KB
_shared/contracts/re_review/legacy/v1_0/input_manifest.schema.jsonconfig6.2 KB
_shared/contracts/re_review/legacy/v1_0/precommitment.schema.jsonconfig8.9 KB
_shared/contracts/re_review/legacy/v1_0/traceability.schema.jsonconfig34.2 KB
_shared/contracts/re_review/legacy/v1_0/verdict_record.schema.jsonconfig10.5 KB
_shared/contracts/re_review/precommitment.schema.jsonconfig8.6 KB
_shared/contracts/re_review/traceability.schema.jsonconfig37.7 KB
_shared/contracts/re_review/verdict_record.schema.jsonconfig10.5 KB
_shared/contracts/research_workflow/inquiry_branch_ledger.schema.jsonconfig17.0 KB
_shared/contracts/research_workflow/research_workflow_profile.schema.jsonconfig7.2 KB
_shared/contracts/research_workflow/research_workflow_profile_selection_receipt.schema.jsonconfig3.9 KB
_shared/contracts/review_target/constructive_review_findings.schema.jsonconfig7.0 KB
_shared/contracts/review_target/criteria_registry.schema.jsonconfig4.0 KB
_shared/contracts/review_target/review_criteria_binding_manifest.schema.jsonconfig7.9 KB
_shared/contracts/review_target/review_criteria_source_receipt.schema.jsonconfig5.6 KB
_shared/contracts/review_target/review_target_context.schema.jsonconfig4.5 KB
_shared/contracts/review_target/review_target_declaration.schema.jsonconfig4.6 KB
_shared/contracts/reviewer/full.jsonconfig3.7 KB
_shared/contracts/reviewer/methodology_focus.jsonconfig2.2 KB
_shared/contracts/reviewer/review_panel_provenance.schema.jsonconfig7.8 KB
_shared/contracts/reviewer/review_panel_provenance_carrier.schema.jsonconfig4.1 KB
_shared/contracts/reviewer/review_panel_provenance_input.schema.jsonconfig3.5 KB
_shared/contracts/revision/author_adjudication.schema.jsonconfig7.8 KB
_shared/contracts/revision/author_adjudication_input.schema.jsonconfig1.2 KB
_shared/contracts/revision/claim_strength_drift_disposition.schema.jsonconfig3.6 KB
_shared/contracts/revision/claim_strength_drift_disposition_input.schema.jsonconfig3.1 KB
_shared/contracts/revision/claim_strength_drift_findings.schema.jsonconfig4.2 KB
_shared/contracts/revision/claim_surface_manifest.schema.jsonconfig3.3 KB
_shared/contracts/revision/integrity_correction_authorization.schema.jsonconfig4.3 KB
_shared/contracts/revision/integrity_correction_authorization_input.schema.jsonconfig1.5 KB
_shared/contracts/revision/integrity_correction_list.schema.jsonconfig1.8 KB
_shared/contracts/revision/integrity_pass_receipt.schema.jsonconfig809 B
_shared/contracts/revision/revision_evidence_bundle.schema.jsonconfig4.9 KB
_shared/contracts/revision/revision_roadmap.schema.jsonconfig11.2 KB
_shared/contracts/submission/format_profile.example.yamlconfig971 B
_shared/contracts/submission/format_profile.schema.jsonconfig4.6 KB
_shared/contracts/submission/submission_verification_report.schema.jsonconfig10.3 KB
_shared/contracts/submission/venue_profile.schema.jsonconfig4.1 KB
_shared/contracts/writer/full.jsonconfig3.6 KB
_shared/cross_model_verification.mdmarkdown83.3 KB
_shared/description-original.mdmarkdown1.1 KB
_shared/dsh-runtime.mdmarkdown1.5 KB
_shared/evals_lift_report.schema.jsonconfig5.0 KB
_shared/ground_truth_isolation_pattern.mdmarkdown16.0 KB
_shared/handoff_schemas.mdmarkdown96.9 KB
_shared/human_subjects_authority_registry.jsonconfig49.5 KB
_shared/mode_spectrum.mdmarkdown5.2 KB
_shared/model_tiering.mdmarkdown12.6 KB
_shared/output_language_pair.mdmarkdown7.7 KB
_shared/policy_data/nature_policy.mdmarkdown3.9 KB
_shared/prisma_trAIce_protocol.mdmarkdown10.2 KB
_shared/raise_framework.mdmarkdown7.5 KB
_shared/references/authority_content_coverage_advisory_protocol.mdmarkdown14.0 KB
_shared/references/claim_standing_candidate_ledger_protocol.mdmarkdown19.2 KB
_shared/references/claim_strength_ladder.mdmarkdown5.5 KB
_shared/references/cross_document_consistency_advisory_protocol.mdmarkdown13.5 KB
_shared/references/evidence_row_protocol.mdmarkdown14.4 KB
_shared/references/firm_rules.mdmarkdown11.0 KB
_shared/references/human_subjects_authority_protocol.mdmarkdown13.1 KB
_shared/references/intent_clarification_protocol.mdmarkdown11.9 KB
_shared/references/irb_terminology_glossary.mdmarkdown12.4 KB
_shared/references/per_source_method_weaknesses.mdmarkdown4.2 KB
_shared/references/protected_hedging_phrases.mdmarkdown8.9 KB
_shared/references/psychometric_terminology_glossary.mdmarkdown8.7 KB
_shared/references/review_criteria_consumer_protocol.mdmarkdown10.3 KB
_shared/references/review_form_note.mdmarkdown11.3 KB
_shared/references/review_pathway_rule_trace_protocol.mdmarkdown8.3 KB
_shared/references/routing_core.mdmarkdown5.7 KB
_shared/references/submission_packet_manifest_protocol.mdmarkdown13.6 KB
_shared/references/word_count_conventions.mdmarkdown9.0 KB
_shared/research_workflow_profiles/field_general.jsonconfig1.2 KB
_shared/review_criteria_registry.jsonconfig11.5 KB
_shared/review_criteria_sources/msr-2027-technical-papers.2026-08-24.jsonconfig2.0 KB
_shared/review_criteria_sources/sigsoft-empirical-standards.2026-08-24.jsonconfig1.4 KB
_shared/sprint_contract.schema.jsonconfig19.0 KB
_shared/style_calibration_protocol.mdmarkdown6.9 KB
_shared/templates/codex_audit_multifile_template.mdmarkdown16.7 KB
agents/protocol_architect_agent.mdmarkdown6.4 KB
agents/qc_auditor_agent.mdmarkdown5.2 KB
agents/reporter_agent.mdmarkdown3.7 KB
agents/screening_reviewer_agent.mdmarkdown5.4 KB
examples/example_config_dta.jsonconfig1.8 KB
examples/example_protocol_dta.mdmarkdown3.8 KB
examples/quick_screen_example.mdmarkdown3.0 KB
references/decision_rules.mdmarkdown6.7 KB
references/failure_paths.mdmarkdown4.1 KB
references/orchestration.mdmarkdown12.4 KB
references/protocol_template.mdmarkdown6.2 KB
references/quality_control.mdmarkdown6.0 KB
references/reporting_and_handoff.mdmarkdown5.8 KB
references/reviewer_roles.mdmarkdown4.0 KB
scripts/_claim_audit_constants.pyscript13.7 KB
scripts/_markdown_lint_util.pyscript9.0 KB
scripts/_next_verified_at_ms.pyscript6.5 KB
scripts/_skill_lint.pyscript13.6 KB
scripts/ars_cache_invalidate.pyscript1.3 KB
scripts/ars_mark_read.pyscript18.2 KB
scripts/ars_write_scope_guard.pyscript30.9 KB
scripts/audit_snapshot.pyscript23.5 KB
scripts/bibliographic_integrity_signals.pyscript32.2 KB
scripts/build_claim_standing_candidate_ledger.pyscript54.3 KB
scripts/build_claim_standing_query_plan.pyscript25.0 KB
scripts/build_content_coverage_advisory.pyscript45.1 KB
scripts/build_cross_document_consistency_advisory.pyscript77.9 KB
scripts/build_outputs.pyscript33.0 KB
scripts/build_review_pathway_rule_trace.pyscript34.2 KB
scripts/build_submission_packet_manifest.pyscript88.1 KB
scripts/build_workflow.pyscript20.0 KB
scripts/check_audit_artifact_consistency.pyscript103.9 KB
scripts/check_benchmark_report.pyscript2.4 KB
scripts/check_claim_audit_consistency.pyscript68.9 KB
scripts/check_claim_standing_freshness.pyscript9.7 KB
scripts/check_claim_standing_transmissions.pyscript17.9 KB
scripts/check_committee_correspondence.pyscript14.9 KB
scripts/check_corpus_consumer_protocol.pyscript13.7 KB
scripts/check_cross_model_handoff_contract.pyscript9.9 KB
scripts/check_data_access_level.pyscript5.2 KB

references/quality_control.md

View on GitHub
> 输出语言(本 DSH 集成强制):所有面向用户的文字、稿件、摘要、评审与报告一律使用**简体中文**;英文术语与引文保持英文。文中的 `zh-TW` / `zh-tw-en` 只是上游语言代码与脚本参数,不代表要用繁体输出。用户明确要求其他语言时以用户为准。
>
> 多角色执行(评审团、十阶段流水线、角色扮演)必须使用 DSH 的 Agent Team 工具(`spawn_teammate`、`team_task_*`、`wait_agent`、`send_message`);当前会话看不到这些工具,说明用户没开 Agent Team,必须先提示用户开启,不得退化成单会话顺序扮演。
# Quality control

## Contents
1. What is checked and when
2. Agreement statistics
3. Seeds
4. Near-miss and random rechecks
5. Human verification and overrides
6. Sources

## 1. What is checked and when

| Check | Pilot | Full run | Blocks progress when |
|-------|-------|----------|----------------------|
| Seed studies found by the search | yes | - | a seed is missing (search problem) |
| Seed studies advanced | yes | yes | a seed is excluded |
| Completeness (no pending records) | yes | yes | always, before reporting |
| Discarded decisions (label/code mismatch, wrong batch) | yes | yes | many in one batch: inspect it |
| Agreement and conflicts | yes | yes | conflicts show a criterion read two ways |
| Unclear share and code distribution | yes | yes | far from the team's expectations |
| Near-miss and random rechecks | - | yes | QC advances a large share of rechecked records |
| Human verification | - | yes | always, before the numbers are published |

## 2. Agreement statistics

Computed on advance (include or unclear) versus exclude, for records decided by both reviewers:

- counts: both advance (a), both exclude (d), A only advances (b), B only advances (c), n = a+b+c+d
- observed agreement po = (a + d) / n
- chance agreement pe = pA*pB + (1 - pA)*(1 - pB), where pA = (a + b)/n, pB = (a + c)/n
- Cohen's kappa = (po - pe) / (1 - pe)
- PABAK = 2*po - 1 (prevalence-adjusted bias-adjusted kappa)

With extreme prevalence, which is normal in screening, kappa is low even when agreement is
excellent: when each reviewer advances only 2-3% of records, 97.8% observed agreement can come
with a kappa near 0.47 and a PABAK of 0.96. Report all four numbers with the counts, and judge
agreement from the conflicts themselves: what the reviewers disagreed about matters more than
the statistic.

## 3. Seeds

Seeds are 3-5 studies the team knows are eligible, listed in the config by DOI, PMID or title.
`prepare_records.py` reports where each one landed after de-duplication:

- **Not found**: the search strategy misses it. Fix the search before screening; screening
  cannot recover a record the search never retrieved.
- **Excluded in the pilot**: read both reasons. Usually a definition is too narrow or a code
  is read too broadly; amend the protocol and pilot again.

Seeds test sensitivity only for studies like them. They do not prove that nothing else was lost.

## 4. Near-miss and random rechecks

- **Near-miss**: an excluded record whose title, abstract or keywords match at least one
  pattern in *every* keyword group of `qc.near_miss`. One group per core criterion keeps the
  list short and on topic: a record that mentions the population, the specimen and the marker,
  yet was excluded, deserves a second look.
- **Joint-exclusion sample (required)**: `qc.random_exclusion_sample` records that both
  reviewers excluded (at least 20, default 100; all of them when fewer), drawn reproducibly
  (`qc.random_seed`) once screening is complete. A joint exclusion never reaches the
  adjudicator, and two instances of the same model can share one misreading, so this sample is
  the only independent look those records get. It catches errors that keywords cannot predict.
- Candidates are packed into their own batch files (`batches/qNNN.txt`, listed in
  `qc_batches.json`), so rechecking a few hundred records takes a handful of agent calls. The
  registry only grows and the joint-exclusion sample is drawn once per review, so repeated
  merges never reshuffle what a QC run referred to. Candidates count as pending until the
  senior reviewer has decided them, so no methods text or final number is produced before,
  including at the full-text stage. Full-text preparation waits for these decisions because a
  QC advance changes which PDFs must be screened.
- A senior reviewer (default model `sonnet`) screens these records afresh without seeing the
  earlier decision. With `qc.policy: advance`, an exclusion it would advance becomes an advance
  marked `QC`; with `flag`, it is only listed.
- Many advances among random rechecks mean the exclusions are not reliable: find the pattern
  (a code, a batch, records without abstracts) before trusting the run.

## 5. Human verification and overrides

AI screening is decision support. Before publication the team should, at minimum:

- check every advanced record (the full-text stage does this anyway), and
- read a random sample of exclusions (50-100 is common; widen the check if an eligible study
  turns up), plus the QC-flagged records.

Team decisions go into `overrides.csv` and win over every automatic decision. Synthetic examples:

```csv
id,d,code,why,by
R00002,unclear,UNC,children after cardiac surgery; urinary marker not named,HUMAN:EXAMPLE
R00003,exclude,E2,piglet model only,HUMAN:EXAMPLE
```

Merge again with `--overrides overrides.csv`. The earlier automatic decision is kept on the
record (`pre_override`) and shown in the log, so the change stays visible.

## 6. Sources

- Page MJ, et al. The PRISMA 2020 statement: an updated guideline for reporting systematic
  reviews. BMJ 2021;372:n71 (item 8, selection process).
- Lefebvre C, et al. Searching for and selecting studies. Cochrane Handbook for Systematic
  Reviews of Interventions, chapter 4.
- Feinstein AR, Cicchetti DV. High agreement but low kappa: I. The problems of two paradoxes.
  J Clin Epidemiol 1990;43:543-9.
- Byrt T, Bishop J, Carlin JB. Bias, prevalence and kappa. J Clin Epidemiol 1993;46:423-9.
add-product-e2e-evalpaperclipai/paperclipAdd or extend a Paperclip full-stack runner E2E workflow, fixture, matcher, or report evidence path for local or Daytona execution.Stars: 115kUpdated Claude CodeCodexVerifiedInstall
add-runner-evalpaperclipai/paperclipAdd or extend a Paperclip Runner protocol evaluation definition, roster, assertion, or report fixture with provenance and narrow validation.Stars: 115kUpdated Claude CodeCodexVerifiedInstall
diagnose-why-work-stoppedpaperclipai/paperclipCompatibility entry point for failed, stalled, looping, or over-recovered Paperclip work. Use fix-failed-paperclip-run for the full diagnosis, regression, repair, real verification, and green-PR workflow; honor an explicitly requested diagnosis-only scope.Stars: 115kUpdated Claude CodeCodexVerifiedInstall
fix-failed-paperclip-runpaperclipai/paperclipDiagnose and fix failed, stuck, slow, or incorrectly blocked Paperclip tasks and runs. Use with a task/run URL or failure report to trace the root cause, prove a regression red then green, check second-order effects, verify the real workflow, and deliver a PR with passing checks.Stars: 115kUpdated Claude CodeCodexVerifiedInstall