aiopshwang/data-analysis-ml-agent-skills / skills/designing-leakage-safe-experiments
designing-leakage-safe-experiments
VerifiedDesign leakage-safe machine learning experiments that mirror real deployment and support fair model comparisons. Use when defining prediction timing, feature eligibility, train-validation-test splits, baselines, metrics, or controlled model iterations; not for auditing whether raw labels are trustworthy.
| File | Kind | Size |
|---|---|---|
| SKILL.md | skill md | 3.8 KB |
| agents/openai.yaml | config | 264 B |
| assets/leakage-safe-experiment-plan.md | markdown | 2.9 KB |
| references/leakage-threats-and-splits.md | markdown | 4.5 KB |
assets/leakage-safe-experiment-plan.md
View on GitHub# Leakage-safe experiment plan
Complete this protocol before model tuning. Amendments made after results are visible belong in the change log.
## Prediction contract
| Field | Definition | Evidence or owner |
|---|---|---|
| Decision and model output | | |
| Entity and prediction grain | | |
| Eligible population at prediction | | |
| Prediction timestamp | | |
| Feature cutoff | | |
| Outcome and maturity window | | |
| Deployment cadence and horizon | | |
| Error costs and capacity constraints | | |
## Feature availability ledger
| Feature or family | Source event | Event time | Typical availability delay | Corrections or backfills | Earliest production availability | Outcome-dependent? | Eligible? | Evidence |
|---|---|---|---|---|---|---|---|---|
| | | | | | | | | |
## Split protocol
- Deployment claim represented:
- Split type and rationale:
- Split unit:
- Train interval or groups:
- Validation interval or groups:
- Final test interval or groups:
- Gap or embargo:
- Duplicate and dependence controls:
- Test access rule:
## Pipeline boundaries
| Step | Learned from training fold only? | Implementation boundary | Persisted artifact | Leakage check |
|---|---|---|---|---|
| Imputation | | | | |
| Encoding or vocabulary | | | | |
| Scaling or normalization | | | | |
| Feature selection | | | | |
| Sampling or weighting | | | | |
| Calibration | separate split or out-of-fold predictions | | | |
| Threshold / decision policy | validation or calibration evidence; never final test | | | |
## Locked comparison
- Simple baseline and why it is credible:
- Primary metric:
- Guardrail metrics:
- Operating threshold or decision rule:
- Uncertainty or variation estimate:
- Required slices:
- Minimum worthwhile improvement:
- Compute or latency budget:
## Planned falsification and stress tests
| Threat | Test capable of revealing it | Pass condition | Result | Disposition |
|---|---|---|---|---|
| Future information | | | | |
| Cross-fold dependence | | | | |
| Full-data preprocessing | | | | |
| Cohort or label maturity | | | | |
| Suspicious feature proxy | | | | |
| Temporal or group instability | | | | |
## Experiment and failure log
| Run ID | Hypothesis | One material change | Frozen data/split ID | Result vs baseline | Slice or stress result | Failure or anomaly | Keep, reject, or investigate |
|---|---|---|---|---|---|---|---|
| | | | | | | | |
## Protocol change log
| Time | Change | Evidence available before change | Reason | Approval if material | New experiment family ID |
|---|---|---|---|---|---|
| | | | | | |
## Final claim gate
- Untouched evidence used for final claim:
- Result and uncertainty:
- Improvement over baseline:
- Conditions where the claim holds:
- Known failure modes:
- Unresolved leakage risks:
- Reproduction command and artifact identifiers:
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| rea-analysis-optimizationmorluto/reaOptimize REA providers and shared analysis workflows for agent context, latency, CPU, and memory while preserving correctness and Evidence. Use for analysis bottlenecks and cache or query design; CI/test-suite optimization belongs to rea-ci-tests-audit. | Stars: 472k | Updated | Claude CodeCodex | Verified | Install |
| rea-ci-tests-auditmorluto/reaAudit REA CI and test suites for correctness, fixture drift, false passes, and measured performance; use for suite or workflow investigations and efficiency changes, rather than ordinary feature testing or a single test failure. | Stars: 472k | Updated | Claude CodeCodex | Verified | Install |
| rea-tool-designmorluto/reaAudit, investigate, design, or change REA investigation tools, CLI/MCP contracts, provider capabilities, and Evidence semantics. | Stars: 472k | Updated | Claude CodeCodex | Verified | Install |
| ponytail-auditDietrichGebert/ponytailQuality audit of the whole repo: bugs, security, real load, missing tests, speed, and what to delete. Most important first. | Stars: 194k | Updated | Claude CodeCodex | Verified | Install |