K-Dense-AI/scientific-agent-skills / skills/clinical-decision-support
clinical-decision-support
VerifiedPrepares and validates research-only clinical decision-support evaluation, evidence-profile, cohort, survival, biomarker/model, privacy, and governance artifacts. Supports aggregate or synthetic research documentation and traceability, excluding patient care and live clinical operation.
references/model_biomarker_evaluation.md
View on GitHub# Aggregate Model and Biomarker Evaluation
## Boundary
Evaluate a locked model, assay, or prespecified biomarker rule using synthetic or aggregate validation summaries. Do not:
- ingest individual records;
- discover or optimize a threshold;
- assign a class or risk to a person;
- match a person to a test or intervention;
- state clinical validity, utility, safety, or fitness for deployment.
## Define the Target
Record:
- evaluation target and version;
- biomarker category using FDA-NIH BEST terminology;
- intended research purpose;
- target population, setting, prevalence, and outcome horizon;
- intended user and non-clinical decision role;
- inputs, output, threshold, and threshold provenance;
- development, tuning, internal-test, and external-validation datasets;
- whether evaluation is temporal, geographic, site-based, or population-based.
Do not call a random split from one source “external validation.”
## Analytical and Clinical Questions
Keep separate:
1. **Analytical validation** — measurement accuracy, precision, detection limits, reproducibility, interference, specimen stability.
2. **Clinical validation** — association or predictive performance for the defined context.
3. **Clinical utility** — whether use improves meaningful outcomes compared with alternatives.
The bundled evaluator addresses only selected aggregate clinical-validation performance summaries. It cannot establish any of the three.
## Performance Dimensions
### Discrimination
Depending on the target:
- sensitivity and specificity;
- predictive values, with prevalence/context;
- likelihood ratios;
- C statistic/AUC with uncertainty;
- time-dependent discrimination for censored outcomes.
Do not report accuracy alone when classes are imbalanced.
### Calibration
Calibration asks whether predicted probabilities agree with observed frequencies. Evaluate:
- calibration-in-the-large;
- calibration slope;
- calibration plots with uncertainty;
- observed versus predicted risk across meaningful ranges;
- integrated or absolute calibration error when justified.
The standard-library helper accepts calibration bins and reports a weighted absolute gap. Its `calibration_in_the_large_gap` field is mean predicted probability minus observed event proportion on the probability scale; it is not the logistic calibration intercept. Bin totals and observed events must match the same group and confusion matrix, including when disclosure suppression applies. Binning loses information and does not replace individual-level calibration analysis in an approved environment.
Primary overview: [Van Calster et al., calibration](https://pubmed.ncbi.nlm.nih.gov/31842878).
### Overall Accuracy and Utility
Use proper scoring rules and decision-curve/net-benefit methods only with a prespecified, defensible decision context. Do not imply utility from AUC or accuracy. Utility evaluation that changes care is outside this skill.
### Uncertainty
Report intervals for performance estimates and explain resampling or analytic methods. A zero denominator is undefined, not zero performance: the helper emits `null` estimates/interval endpoints. Balanced accuracy requires both observed outcome classes, otherwise it is `null` and excluded from subgroup differences. The helper uses Wilson intervals for aggregate proportions. It does not model correlated observations, clustering, repeated measurements, censoring, or verification bias.
## External Validation
Apply the original locked model without refitting. Document:
- differences in case mix, prevalence, setting, workflow, and measurement;
- eligibility and missingness;
- sample-size rationale based on precision targets, not a blanket event rule;
- calibration, discrimination, and any prespecified utility measure;
- subgroup performance;
- model failures and unusable inputs;
- whether recalibration was separate from validation.
See [BMJ 2024 external-validation guidance](https://www.bmj.com/content/384/bmj-2023-074820) and [sample-size methodology](https://pmc.ncbi.nlm.nih.gov/articles/PMC8352630).
## Subgroup and Fairness Evaluation
Before analysis:
- identify groups based on intended use, evidence, and stakeholder input;
- document category provenance and limitations;
- set minimum precision and disclosure rules;
- plan intersectional analyses where feasible;
- define metrics and acceptable uncertainty;
- evaluate measurement and label validity;
- plan investigation and mitigation, not only detection.
Report:
- representation and missingness;
- performance and calibration with intervals;
- data quality and failure rates;
- distribution shift;
- human-AI interaction where relevant;
- observed differences without declaring a group deficient.
No single parity metric establishes fairness. Equal metrics can coexist with inequitable outcomes, and unequal metrics may reflect case mix, measurement, structural conditions, or model behavior that requires investigation.
## Biomarker-Specific Controls
- Pre-specify specimen, assay, platform, software, quality controls, units, and threshold.
- Preserve continuous information where appropriate.
- Separate prognostic association from treatment-effect interaction.
- Blind assay assessment to outcome when feasible.
- Account for batch/site effects and failed measurements.
- Validate thresholds independently.
- Report analytical validity before clinical interpretation.
- Use REMARK for tumor prognostic markers.
## Change Control and Monitoring
For each release record:
- immutable model/assay version;
- data and code versions;
- planned changes and rationale;
- validation protocol and acceptance criteria;
- subgroup/calibration regression tests;
- human-factors impact;
- approval and rollback;
- monitoring cadence and drift triggers;
- incident handling and retirement.
Never update a threshold or model silently after viewing performance.
## Aggregate Evaluator
Input contains only:
- group labels and aggregate denominators;
- confusion counts;
- aggregate calibration bins;
- provenance and validation metadata.
Output contains bounded descriptive metrics, uncertainty, suppression, and documented gaps. Invalid inputs return failure diagnostics with no partial group metrics. The helper emits JSON only; it does not produce calibration plots, bin-level Wilson intervals, forest plots, or fitted calibration models. It never outputs a person-level class or recommendation.
```bash
python3 scripts/model_biomarker_evaluation.py \
assets/aggregate_model_evaluation_template.json
```
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| rea-changelog-updatemorluto/reaPrepare or rewrite REA release changelogs and GitHub release notes from pinned Git history, with verified contributor thanks and Release Please synchronization. | Stars: 472k | Updated | Claude CodeCodex | Verified | Install |
| academy-guideanthropics/skillsStop and check this skill before finishing any reply to a question about how to use Claude or a Claude product — it recommends matching courses, tutorials, and use cases from Claude Academy (academy.claude.com), Anthropic's learning hub. Trigger on: "how do I", "how can I", "getting started with", "what can Claude do", "teach me", "learn to use"; questions about artifacts, projects, skills, plugins, connectors, MCP; requests about rolling Claude out to a team, class, or organization; and any ask for training materials, onboarding content, or learning resources. Use it when the user is learning how to use a feature or product — not when they are mid-task and just want the task done. This skill composes with other skills: after consulting product documentation to answer how a Claude feature works, also check here for a matching course or tutorial — a docs-grounded answer and an Academy recommendation belong together. Only recommend on a strong match; never invent Academy content. | Stars: 198k | Updated | Claude CodeCodex | Verified | Install |
| claude-apianthropics/skillsReference for the Claude API / Anthropic SDK — model ids, pricing, params, streaming, tool use, MCP, agents, caching, token counting, model migration. TRIGGER — read BEFORE opening the target file; don't skip because it "looks like a one-liner" — whenever: the prompt names Claude/Anthropic in any form (Claude, Anthropic, Fable, Opus, Sonnet, Haiku, `anthropic`, `@anthropic-ai`, `claude-*`, `us.anthropic.*`, `[1m]`); the user asks about an LLM (pricing/model choice/limits/caching) — never answer from memory; OR the task is LLM-shaped with provider unstated (agent/MCP/tool-definition/multi-agent/RAG/LLM-judge/computer-use; generate/summarize/extract/classify/rewrite/converse over NL; debugging refusals/cutoffs/streaming/tool-calls/tokens). SKIP only when another provider is being worked on (overrides all triggers): OpenAI/GPT/Gemini/Llama/Mistral/Cohere/Ollama named in the query; OR `grep -rE 'openai|langchain_openai|google.generativeai|genai|mistralai|cohere|ollama'` over the project hits (run this grep FIRST if no provider named — don't Read the file). | Stars: 198k | Updated | Claude CodeCodex | Unverified | Install |
| doc-coauthoringanthropics/skillsGuide users through a structured workflow for co-authoring documentation. Use when user wants to write documentation, proposals, technical specs, decision docs, or similar structured content. This workflow helps users efficiently transfer context, refine content through iteration, and verify the doc works for readers. Trigger when user mentions writing docs, creating proposals, drafting specs, or similar documentation tasks. | Stars: 198k | Updated | Claude CodeCodex | Verified | Install |