Skip to content

aiopshwang/data-analysis-ml-agent-skills / skills/validating-models-and-claims

validating-models-and-claims

Verified

Validate trained models and analytical claims against their intended decision, independent evidence, and human-reviewed ground truth. Use when reviewing model performance, analysis conclusions, launch claims, or evaluation reports; use failure diagnosis instead when the main task is locating a known defect.

FileKindSize
SKILL.mdskill md4.6 KB
agents/openai.yamlconfig261 B
assets/validation-report.mdmarkdown3.0 KB
references/validation-protocol.mdmarkdown4.9 KB

assets/validation-report.md

View on GitHub
# Model and Claim Validation Report

## Decision summary

- **Artifact or analysis:**
- **Artifact version / commit:**
- **Decision this review informs:**
- **Validation date:**
- **Validator(s) and independence:**
- **Overall recommendation:** approve / approve with conditions / reject / evidence incomplete
- **Supported operating scope:**
- **Conditions or owner actions:**

## Claim-to-evidence matrix

| ID | Falsifiable claim | Population / unit / horizon | Acceptance criterion | Evidence ID | Status | Supported wording |
|---|---|---|---|---|---|---|
| C-01 |  |  |  | E-01 | not tested |  |

Allowed status: `supported`, `partially supported`, `unsupported`, `not tested`.

## Evidence register

| ID | Source / snapshot | Used during development? | Independent owner? | Protocol frozen? | Operational path? | Limitations |
|---|---|---:|---:|---:|---:|---|
| E-01 |  |  |  |  |  |  |

## Evaluation contract

- **Target definition and observation window:**
- **Prediction time and feature cutoff:**
- **Sampling and split boundary:**
- **Comparator:**
- **Operating threshold and how chosen:**
- **Primary metric, denominator, and aggregation unit:**
- **Practical improvement threshold:**
- **Uncertainty method:**
- **Evaluation-set contact log:**

## Human ground truth

- **Why human review is or is not required:**
- **Sampling frame and sample size:**
- **Reviewer qualifications and blinding:**
- **Instructions / label version:**
- **Agreement and disagreement:**
- **Adjudication method:**
- **Unresolved or ambiguous cases:**

## Results

| Cohort or slice | N / positives | Candidate | Comparator | Difference | Interval | Criterion met? |
|---|---:|---:|---:|---:|---|---:|
| Overall |  |  |  |  |  |  |

State exclusions, missingness, and multiple-comparison caveats next to affected results.

## Error review

| Case ID | Expected | Observed | Cost / impact | Human review | Likely layer | Follow-up |
|---|---|---|---|---|---|---|
|  |  |  |  |  |  |  |

- **Sampling method for reviewed errors:**
- **Recurring error modes:**
- **High-cost or safety-relevant cases:**

## Operational fidelity

| Check | Evidence | Result | Gap / owner |
|---|---|---|---|
| Feature availability at decision time |  |  |  |
| Offline/online transformation parity |  |  |  |
| Artifact, class order, and threshold |  |  |  |
| Output interpretation and fallback |  |  |  |

## Contradictory evidence and rejected interpretations

Record results that disagree with the preferred conclusion and why each alternative interpretation was rejected or retained.

| Observation | Interpretation considered | Evidence for / against | Disposition |
|---|---|---|---|
|  |  |  |  |

## Limitations and missing proof

- **Untested populations or conditions:**
- **Known label or sampling limitations:**
- **Evidence that would change the decision:**
- **Next validation and owner:**

## Sign-off

| Role | Name | Decision | Date | Conditions acknowledged |
|---|---|---|---|---|
| Validation owner |  |  |  |  |
| Decision owner |  |  |  |  |
receiving-code-reviewobra/superpowersUse when receiving code review feedback, before implementing suggestions, especially if feedback seems unclear or technically questionable - requires technical rigor and verification, not performative agreement or blind implementationStars: 322kUpdated Claude CodeCodexVerifiedInstall
requesting-code-reviewobra/superpowersUse when completing tasks, implementing major features, or before merging to verify work meets requirementsStars: 322kUpdated Claude CodeCodexVerifiedInstall
using-git-worktreesobra/superpowersUse when starting feature work that needs isolation from current workspace or before executing implementation plans - ensures an isolated workspace exists via native tools or git worktree fallbackStars: 322kUpdated Claude CodeCodexVerifiedInstall
verification-before-completionobra/superpowersUse when about to claim work is complete, fixed, or passing, before committing or creating PRs - requires running verification commands and confirming output before making any success claims; evidence before assertions alwaysStars: 322kUpdated Claude CodeCodexVerifiedInstall