Skip to content

sb-dev/openquant / .agents/skills/ml4t-point-in-time

ml4t-point-in-time

Verified

Ensure data reflects what was known at each decision point, not revised or restated values. Use when joining fundamental, macro, or alternative data to price series.

Point-in-Time Correctness

Joining data by event date instead of availability date uses information that did not exist yet, inflating backtest returns by 1-3% annually on fundamental strategies.

The Problem

Fundamental data has two timestamps: when the economic event occurred (quarter-end) and when the data became publicly available (SEC filing date, press release, data vendor publication). Using the event date treats revised, delayed, or embargoed data as if it were available in real time.

Example: Apple's Q4 2024 revenue (period ending Dec 31) is filed with the SEC on Jan 31, 2025. A model that joins revenue on Dec 31 uses data 31 days before it existed.

Macro data has the same problem: GDP is released ~30 days after quarter-end and revised two to three times over the following months.

The Pattern

WRONG

import polars as pl

# Join fundamentals on the quarter they describe
prices = pl.read_parquet("prices.parquet")
fundamentals = pl.read_parquet("fundamentals.parquet")

df = prices.join(
    fundamentals,
    left_on=["symbol", "timestamp"],
    right_on=["symbol", "quarter_end"],    # uses event date
    how="left",
)

CORRECT

import polars as pl

prices = pl.read_parquet("prices.parquet")
fundamentals = pl.read_parquet("fundamentals.parquet")

# Join on filing date (when the data became publicly available)
df = prices.join_asof(
    fundamentals.sort("filing_date"),
    left_on="timestamp",
    right_on="filing_date",              # uses availability date
    by="symbol",
    strategy="backward",                 # only use data available by this date
)

Key Date Types

Date fieldWhat it meansSafe to join on?
quarter_end / period_endWhen the economic event occurredNo
filing_date / published_atWhen the data became publicYes
release_date (macro)When the statistical agency published itYes
revised_dateWhen a correction was issuedOnly for the revision

Macro Data: Release Calendars

# GDP example: align by release date, not observation quarter
gdp_releases = pl.DataFrame({
    "observation_quarter": ["2024-Q3", "2024-Q3", "2024-Q3"],
    "release_date": ["2024-10-30", "2024-11-27", "2024-12-19"],
    "release_type": ["advance", "second", "third"],
    "value": [4.9, 5.2, 4.9],
})
# A point-in-time feature on Nov 1, 2024 should use 4.9 (advance), not 5.2

Guardrails

  • Every fundamental join must use a filing_date or release_date column, never quarter_end or period_end.
  • For SEC data, use accepted_at (filing acceptance timestamp), not the cover-page period date. Vendor "current" snapshots are NOT point-in-time safe.
  • FRED data: check the release calendar (FRED/releases), not the observation date.
  • Earnings data: available after market close on announcement day, not at open.
  • Add a 1-day buffer after filing date to account for data vendor processing lag.

Production Implementation

ml4t-data provides point-in-time access for specific datasets rather than a generic PIT join helper:

from ml4t.data.providers.fred import FREDProvider

provider = FREDProvider()
unrate = provider.fetch_ohlcv(
    "UNRATE",
    "2024-01-01",
    "2024-03-31",
    vintage_date="2024-03-15",
)
# Multi-dataset PIT joins still belong in your research code

Checklist

  • All fundamental joins use filing_date / release_date, not period-end
  • Macro features aligned by publication date with release calendar
  • 1-day buffer added for data processing lag
  • No "as-reported" vs "revised" confusion in the feature pipeline
  • join_asof with strategy="backward" used for temporal alignment
slack-gif-creatoranthropics/skillsKnowledge and utilities for creating animated GIFs optimized for Slack. Provides constraints, validation tools, and animation concepts. Use when users request animated GIFs for Slack like "make me a GIF of X doing Y for Slack."Stars: 196kUpdated Claude CodeCodexVerifiedInstall
ponytailDietrichGebert/ponytailLazy senior dev mode for any coding task: the smallest change that fully solves it, and a reply a busy human understands in one read.Stars: 192kUpdated Claude CodeCodexVerifiedInstall
ponytail-debtDietrichGebert/ponytailHarvest every shortcut: comment into one debt ledger, so deferrals get tracked instead of forgotten. One-shot report.Stars: 192kUpdated Claude CodeCodexVerifiedInstall
ponytail-debtDietrichGebert/ponytailList every `shortcut:` comment (and older `ponytail:` ones) as a debt ledger. One-shot report, changes nothing. Use for "ponytail debt", "what did ponytail defer", "list the shortcuts", /ponytail-debt.Stars: 192kUpdated Claude CodeCodexVerifiedInstall