sb-dev/openquant / .agents/skills/ml4t-validate-data
ml4t-validate-data
VerifiedValidate financial data for schema errors, missing sessions, duplicates, corporate actions and provenance before research or evaluation.
Validate financial data
Read OQ-02 §§1.1 and 3 and OQ-05 §7 in docs/specs/. Validate with OpenQuant contracts; do not adopt a second Python schema or numerical engine.
Check the dataset
- Verify schema/version, stable instrument identity, units, currency, timezone/session, observation order and requested versus returned coverage.
- Check finite values, precision, high/low relationships and open/close bounds where the instrument/bar contract requires them. Keep decimal transformations consistent with the Rust core.
- Detect duplicate identities. Identical retransmissions may be deduplicated under a declared rule; conflicting observations require explicit revision handling. Preserve the originals and the decision.
- Compare expected sessions with observed bars. Distinguish holidays, early closes, halts, no qualifying trades, missing pages and source outages. Never infer completeness solely from row count.
- Check declared split/dividend adjustments against corporate-action records. Investigate extreme moves and repeated prices; neither is automatically a data error. Do not erase valid adverse returns or invent delisting proceeds.
- Check event, publication, availability and retrieval times separately. Preserve unknown times and timestamp precision. Exclude evidence that cannot satisfy the registered cutoff policy.
Decide and record
Separate invalid structure from quality findings needing investigation. Apply predeclared thresholds and explain exclusions. Missing required evidence cannot pass. Do not repair an official dataset silently; produce a new version with provenance. Forward-fill is allowed only for explicitly defined derived features with freshness limits, never as invented trades, fills or executable prices.
Verify
Use small independent fixtures for split/dividend treatment, valid large moves, halts, conflicting duplicates, missing sessions and revised records. Run validation on every new snapshot. Report findings and affected research eligibility. A provider agreement check is not an independent numerical baseline.
Adapted from ML4T skills at revision 691857537a0c8befc11dc2f2ad7a458b7c7e801c. Original Apache-2.0 licence and attribution remain in .claude/third-party/ml4t/skills. OpenQuant adaptation: 2026-11-05.
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| add-product-e2e-evalpaperclipai/paperclipAdd or extend a Paperclip full-stack runner E2E workflow, fixture, matcher, or report evidence path for local or Daytona execution. | Stars: 114k | Updated | Claude CodeCodex | Verified | Install |
| add-runner-evalpaperclipai/paperclipAdd or extend a Paperclip Runner protocol evaluation definition, roster, assertion, or report fixture with provenance and narrow validation. | Stars: 114k | Updated | Claude CodeCodex | Verified | Install |
| diagnose-why-work-stoppedpaperclipai/paperclipCompatibility entry point for failed, stalled, looping, or over-recovered Paperclip work. Use fix-failed-paperclip-run for the full diagnosis, regression, repair, real verification, and green-PR workflow; honor an explicitly requested diagnosis-only scope. | Stars: 114k | Updated | Claude CodeCodex | Verified | Install |
| fix-failed-paperclip-runpaperclipai/paperclipDiagnose and fix failed, stuck, slow, or incorrectly blocked Paperclip tasks and runs. Use with a task/run URL or failure report to trace the root cause, prove a regression red then green, check second-order effects, verify the real workflow, and deliver a PR with passing checks. | Stars: 114k | Updated | Claude CodeCodex | Verified | Install |