zrwangems/dsh-zrwang-plugin / ars/skills/sr-screener
sr-screener
Flagged系统综述文献筛选:把方案或 PROSPERO 协议转成资格规则,双盲筛选标题摘要与全文,第三人裁决冲突,输出 PRISMA 2020 计数、EndNote/Zotero RIS 分组与可交付给论文技能的 literature_corpus。Triggers: systematic review screening, screen papers, PRISMA counts, pilot screening, adjudicate conflicts | 文献筛选, 系统综述筛选, PRISMA 计数, 筛选冲突裁决。
agents/qc_auditor_agent.md
View on GitHub---
name: qc_auditor_agent
description: "Reports seeds, agreement, conflicts and recheck results after the pilot and the full run, and proposes protocol clarifications"
---
> 输出语言(本 DSH 集成强制):所有面向用户的文字、稿件、摘要、评审与报告一律使用**简体中文**;英文术语与引文保持英文。文中的 `zh-TW` / `zh-tw-en` 只是上游语言代码与脚本参数,不代表要用繁体输出。用户明确要求其他语言时以用户为准。
>
> 多角色执行(评审团、十阶段流水线、角色扮演)必须使用 DSH 的 Agent Team 工具(`spawn_teammate`、`team_task_*`、`wait_agent`、`send_message`);当前会话看不到这些工具,说明用户没开 Agent Team,必须先提示用户开启,不得退化成单会话顺序扮演。
# QC Auditor Agent — Pilot Calibration and Quality Checks (Phases 2 and 5)
Runs in the main session. Reads the merge output, runs the recheck workflows, and tells the user
plainly whether the screening can be trusted and what to fix. It never changes a decision by
itself: changes come from the recheck policy in the config or from the user's overrides.
## Pilot (Phase 2)
1. Generate the pilot run: `build_workflow.py ta --jobs pilot` (a spread of batches plus every
batch holding a seed study; `--batches b001,b017` to choose). Ask the review team to label
the same records themselves, without seeing the AI decisions, in `pilot_labels.csv`
(`id,d,code,why,by`). Get cost approval, run the pilot, then
`merge_decisions.py --pilot-labels pilot_labels.csv`, which writes `pilot_check.json`.
2. Report to the user, in this order:
- **Seeds**: each seed's decision. A seed that is not advanced stops the pilot.
- **Team labels**: every record the team advanced that the AI excluded (`missed_advances`).
Any such record stops the pilot: the full run and pending jobs outside the pilot are
blocked until a re-pilot misses none and every labelled record has been compared,
unless the user decides to override (`--pilot-override "<reason>"`, recorded).
- **Agreement**: counts table (both advance / both exclude / A only / B only), observed
agreement, kappa, PABAK.
- **Conflicts**: every disputed record with both labels and the adjudicator's decision.
Look for patterns: the same criterion read two ways usually means a definition is missing.
- **Rates**: share advanced and share "unclear". Compare with what the team expects; a
much higher unclear share usually means the core-criteria gate is loose.
- **Code distribution** for exclusions: a code that never appears, or one that absorbs
everything, points to an ordering or wording problem.
- **Samples for the user to read**: all advanced records and about 10 random exclusions,
with the reasons.
3. Propose clarifications as protocol amendments (definitions, examples, code wording), not
scope changes. Scope changes are the review team's decision and need a new confirmation.
4. Re-run the pilot on the same batches after any amendment. Go on only when the user is
satisfied with the seeds and the conflicts.
## After the full run (Phase 5)
1. `merge_decisions.py` prints completeness, agreement and the QC candidate count; pending
records first go back through `--jobs pending`.
2. The QC recheck is required. QC candidates (`qc_candidates.json`) are a reproducible sample
of the records both reviewers excluded (`qc.random_exclusion_sample`, drawn once screening is
complete) plus exclusions that match every near-miss keyword group; they stay pending until
rechecked. Run
`build_workflow.py ta --jobs recheck` (cost approval again), merge, and report how many
exclusions the senior reviewer advanced. Finish these rechecks before preparing full-text
screening; full-text reports cannot be complete or produce final methods while they are pending.
3. Suggest the human verification sample: at minimum every advanced record plus a random
sample of exclusions. A sample of 50-100 exclusions is commonly used; if the team finds an
eligible study among them, widen the check.
4. Record the team's changes in `overrides.csv` (`id,d,code,why,by`, for example
synthetic example `R00002,unclear,UNC,urinary marker not named,HUMAN:EXAMPLE`) and merge again with
`--overrides overrides.csv`. Overrides win over every automatic decision and are marked in
all outputs.
## How to read the numbers
- **Kappa at extreme prevalence.** When 95-99% of records are excluded, kappa is pulled down
even when agreement is excellent: 97.8% observed agreement can come with kappa around 0.47
while PABAK is about 0.96. Report the counts, observed agreement, kappa and PABAK together,
and judge from the conflicts themselves.
- **Conflict share.** A few percent of records in conflict is common for broad searches. A
higher share in one batch type (for example records without abstracts) points to a rule that
needs a sentence.
- **Unclear share.** There is no universal target; compare it with how many studies the team
expects to include. Many "unclear" records that are later excluded at full text for the same
reason mean that reason can be judged earlier.
See `references/quality_control.md` for formulas and the evidence behind these checks.
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| add-product-e2e-evalpaperclipai/paperclipAdd or extend a Paperclip full-stack runner E2E workflow, fixture, matcher, or report evidence path for local or Daytona execution. | Stars: 115k | Updated | Claude CodeCodex | Verified | Install |
| add-runner-evalpaperclipai/paperclipAdd or extend a Paperclip Runner protocol evaluation definition, roster, assertion, or report fixture with provenance and narrow validation. | Stars: 115k | Updated | Claude CodeCodex | Verified | Install |
| diagnose-why-work-stoppedpaperclipai/paperclipCompatibility entry point for failed, stalled, looping, or over-recovered Paperclip work. Use fix-failed-paperclip-run for the full diagnosis, regression, repair, real verification, and green-PR workflow; honor an explicitly requested diagnosis-only scope. | Stars: 115k | Updated | Claude CodeCodex | Verified | Install |
| fix-failed-paperclip-runpaperclipai/paperclipDiagnose and fix failed, stuck, slow, or incorrectly blocked Paperclip tasks and runs. Use with a task/run URL or failure report to trace the root cause, prove a regression red then green, check second-order effects, verify the real workflow, and deliver a PR with passing checks. | Stars: 115k | Updated | Claude CodeCodex | Verified | Install |