aiopshwang/data-analysis-ml-agent-skills / skills/validating-models-and-claims
validating-models-and-claims
VerifiedValidate trained models and analytical claims against their intended decision, independent evidence, and human-reviewed ground truth. Use when reviewing model performance, analysis conclusions, launch claims, or evaluation reports; use failure diagnosis instead when the main task is locating a known defect.
| File | Kind | Size |
|---|---|---|
| SKILL.md | skill md | 4.6 KB |
| agents/openai.yaml | config | 261 B |
| assets/validation-report.md | markdown | 3.0 KB |
| references/validation-protocol.md | markdown | 4.9 KB |
assets/validation-report.md
View on GitHub# Model and Claim Validation Report
## Decision summary
- **Artifact or analysis:**
- **Artifact version / commit:**
- **Decision this review informs:**
- **Validation date:**
- **Validator(s) and independence:**
- **Overall recommendation:** approve / approve with conditions / reject / evidence incomplete
- **Supported operating scope:**
- **Conditions or owner actions:**
## Claim-to-evidence matrix
| ID | Falsifiable claim | Population / unit / horizon | Acceptance criterion | Evidence ID | Status | Supported wording |
|---|---|---|---|---|---|---|
| C-01 | | | | E-01 | not tested | |
Allowed status: `supported`, `partially supported`, `unsupported`, `not tested`.
## Evidence register
| ID | Source / snapshot | Used during development? | Independent owner? | Protocol frozen? | Operational path? | Limitations |
|---|---|---:|---:|---:|---:|---|
| E-01 | | | | | | |
## Evaluation contract
- **Target definition and observation window:**
- **Prediction time and feature cutoff:**
- **Sampling and split boundary:**
- **Comparator:**
- **Operating threshold and how chosen:**
- **Primary metric, denominator, and aggregation unit:**
- **Practical improvement threshold:**
- **Uncertainty method:**
- **Evaluation-set contact log:**
## Human ground truth
- **Why human review is or is not required:**
- **Sampling frame and sample size:**
- **Reviewer qualifications and blinding:**
- **Instructions / label version:**
- **Agreement and disagreement:**
- **Adjudication method:**
- **Unresolved or ambiguous cases:**
## Results
| Cohort or slice | N / positives | Candidate | Comparator | Difference | Interval | Criterion met? |
|---|---:|---:|---:|---:|---|---:|
| Overall | | | | | | |
State exclusions, missingness, and multiple-comparison caveats next to affected results.
## Error review
| Case ID | Expected | Observed | Cost / impact | Human review | Likely layer | Follow-up |
|---|---|---|---|---|---|---|
| | | | | | | |
- **Sampling method for reviewed errors:**
- **Recurring error modes:**
- **High-cost or safety-relevant cases:**
## Operational fidelity
| Check | Evidence | Result | Gap / owner |
|---|---|---|---|
| Feature availability at decision time | | | |
| Offline/online transformation parity | | | |
| Artifact, class order, and threshold | | | |
| Output interpretation and fallback | | | |
## Contradictory evidence and rejected interpretations
Record results that disagree with the preferred conclusion and why each alternative interpretation was rejected or retained.
| Observation | Interpretation considered | Evidence for / against | Disposition |
|---|---|---|---|
| | | | |
## Limitations and missing proof
- **Untested populations or conditions:**
- **Known label or sampling limitations:**
- **Evidence that would change the decision:**
- **Next validation and owner:**
## Sign-off
| Role | Name | Decision | Date | Conditions acknowledged |
|---|---|---|---|---|
| Validation owner | | | | |
| Decision owner | | | | |
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| receiving-code-reviewobra/superpowersUse when receiving code review feedback, before implementing suggestions, especially if feedback seems unclear or technically questionable - requires technical rigor and verification, not performative agreement or blind implementation | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |
| requesting-code-reviewobra/superpowersUse when completing tasks, implementing major features, or before merging to verify work meets requirements | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |
| using-git-worktreesobra/superpowersUse when starting feature work that needs isolation from current workspace or before executing implementation plans - ensures an isolated workspace exists via native tools or git worktree fallback | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |
| verification-before-completionobra/superpowersUse when about to claim work is complete, fixed, or passing, before committing or creating PRs - requires running verification commands and confirming output before making any success claims; evidence before assertions always | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |