Skip to content

sb-dev/openquant / .agents/skills/ml4t-validate-data

ml4t-validate-data

Verified

Validate financial data for schema errors, missing sessions, duplicates, corporate actions and provenance before research or evaluation.

Validate financial data

Read OQ-02 §§1.1 and 3 and OQ-05 §7 in docs/specs/. Validate with OpenQuant contracts; do not adopt a second Python schema or numerical engine.

Check the dataset

  1. Verify schema/version, stable instrument identity, units, currency, timezone/session, observation order and requested versus returned coverage.
  2. Check finite values, precision, high/low relationships and open/close bounds where the instrument/bar contract requires them. Keep decimal transformations consistent with the Rust core.
  3. Detect duplicate identities. Identical retransmissions may be deduplicated under a declared rule; conflicting observations require explicit revision handling. Preserve the originals and the decision.
  4. Compare expected sessions with observed bars. Distinguish holidays, early closes, halts, no qualifying trades, missing pages and source outages. Never infer completeness solely from row count.
  5. Check declared split/dividend adjustments against corporate-action records. Investigate extreme moves and repeated prices; neither is automatically a data error. Do not erase valid adverse returns or invent delisting proceeds.
  6. Check event, publication, availability and retrieval times separately. Preserve unknown times and timestamp precision. Exclude evidence that cannot satisfy the registered cutoff policy.

Decide and record

Separate invalid structure from quality findings needing investigation. Apply predeclared thresholds and explain exclusions. Missing required evidence cannot pass. Do not repair an official dataset silently; produce a new version with provenance. Forward-fill is allowed only for explicitly defined derived features with freshness limits, never as invented trades, fills or executable prices.

Verify

Use small independent fixtures for split/dividend treatment, valid large moves, halts, conflicting duplicates, missing sessions and revised records. Run validation on every new snapshot. Report findings and affected research eligibility. A provider agreement check is not an independent numerical baseline.

Adapted from ML4T skills at revision 691857537a0c8befc11dc2f2ad7a458b7c7e801c. Original Apache-2.0 licence and attribution remain in .claude/third-party/ml4t/skills. OpenQuant adaptation: 2026-11-05.

add-product-e2e-evalpaperclipai/paperclipAdd or extend a Paperclip full-stack runner E2E workflow, fixture, matcher, or report evidence path for local or Daytona execution.Stars: 114kUpdated Claude CodeCodexVerifiedInstall
add-runner-evalpaperclipai/paperclipAdd or extend a Paperclip Runner protocol evaluation definition, roster, assertion, or report fixture with provenance and narrow validation.Stars: 114kUpdated Claude CodeCodexVerifiedInstall
diagnose-why-work-stoppedpaperclipai/paperclipCompatibility entry point for failed, stalled, looping, or over-recovered Paperclip work. Use fix-failed-paperclip-run for the full diagnosis, regression, repair, real verification, and green-PR workflow; honor an explicitly requested diagnosis-only scope.Stars: 114kUpdated Claude CodeCodexVerifiedInstall
fix-failed-paperclip-runpaperclipai/paperclipDiagnose and fix failed, stuck, slow, or incorrectly blocked Paperclip tasks and runs. Use with a task/run URL or failure report to trace the root cause, prove a regression red then green, check second-order effects, verify the real workflow, and deliver a PR with passing checks.Stars: 114kUpdated Claude CodeCodexVerifiedInstall