Skip to content

sb-dev/openquant / .agents/skills/ml4t-lookahead-bias

ml4t-lookahead-bias

Verified

Detect future information leaking into features, labels, or evaluation. Use when any pipeline step might expose data not yet available at prediction time.

Lookahead Bias

The most common ML4T failure. A model uses future information during training - normalization with full-sample statistics, labels from future thresholds, standard k-fold CV on time series - producing a great backtest that fails immediately in production.

The Problem

Lookahead bias means using information that would not have been available at prediction time. It inflates backtest Sharpe and is the #1 reason strategies fail live. Common sources: full-sample normalization, future-based thresholds, train+test preprocessing, and shuffled CV on time series.

The Pattern

WRONG

from sklearn.preprocessing import StandardScaler

# Normalize features using full dataset statistics (future leak)
scaler = StandardScaler()
X_scaled = scaler.fit_transform(X)  # mean/std computed on ALL data including test

# Label threshold from full history (future leak)
threshold = returns.quantile(0.75)  # Uses future returns to set threshold
y = (forward_returns > threshold).astype(int)

# Shuffled k-fold on time series (future leak)
from sklearn.model_selection import KFold
cv = KFold(n_splits=5, shuffle=True)  # Training on 2024 data, testing on 2020

CORRECT

from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import Pipeline

# Expanding-window normalization: only past data
X_normalized = (X - X.expanding().mean().shift(1)) / X.expanding().std().shift(1)

# Expanding threshold: only past returns
threshold = returns.expanding().quantile(0.75).shift(1)
y = (forward_returns > threshold).astype(int)

# Time-series CV that preserves order
from sklearn.model_selection import TimeSeriesSplit
cv = TimeSeriesSplit(n_splits=5)

Preprocessing Pipeline

from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import Ridge

# Pipeline ensures scaler is fit on train only, even inside CV
pipe = Pipeline([
    ("scaler", StandardScaler()),
    ("model", Ridge()),
])
pipe.fit(X_train, y_train)       # scaler sees only training data
pipe.predict(X_test)             # test data transformed with train stats

Detection

  • In-sample and out-of-sample performance are suspiciously similar (gap < 5%)
  • Sharpe ratio > 2.0 on daily data with a simple model
  • Feature values change when you recompute with more recent data appended
  • Performance degrades sharply when switching from k-fold to walk-forward CV
  • Labels use close-to-close returns but execution is next-open - this 50-100 bps gap per trade is hidden lookahead

Guardrails

  • All rolling/expanding statistics must use .shift(1) to exclude the current observation
  • fit_transform() must never touch test data - use Pipeline or manual train/test splits
  • Cross-validation must respect temporal order - no shuffle=True on time series
  • Label thresholds must be computed from past data only (expanding window)
  • Point-in-time fundamentals: use report dates, not period dates

Production Implementation

from ml4t.diagnostic.splitters import CombinatorialCV
from ml4t.engineer import create_dataset_builder

builder = create_dataset_builder(features, labels, dates=timestamps, scaler="standard")
cv = CombinatorialCV(n_groups=8, n_test_groups=2, embargo_pct=0.01)
fold = next(builder.split(cv))
X_train, y_train = fold.X_train, fold.y_train
X_test, y_test = fold.X_test, fold.y_test

Checklist

  • All normalizations use expanding window with .shift(1), not full-sample stats
  • fit_transform() only on training data (use Pipeline or explicit split)
  • Time-series CV with purging and embargo (not shuffled k-fold)
  • Labels do not use future thresholds or statistics
  • Point-in-time data: report dates used, not period-end dates
  • Suspicious results investigated: Sharpe > 2, IS/OOS gap < 5%
writing-skillsobra/superpowersUse when creating new skills, editing existing skills, or verifying skills work before deploymentStars: 319kUpdated Claude CodeCodexVerifiedInstall
xlsxanthropics/skillsUse this skill any time a spreadsheet file is the primary input or output. This means any task where the user wants to: open, read, edit, or fix an existing .xlsx, .xlsm, .xltx, .csv, or .tsv file (e.g., adding columns, computing formulas, formatting, charting, cleaning messy data); create a new spreadsheet from scratch or from other data sources; or convert between tabular file formats. Trigger especially when the user references a spreadsheet file by name or path — even casually (like "the xlsx in my downloads") — and wants something done to it or produced from it. Also trigger for cleaning or restructuring messy tabular data files (malformed rows, misplaced headers, junk data) into proper spreadsheets. The deliverable must be a spreadsheet file. Do NOT trigger when the primary deliverable is a Word document, HTML report, standalone Python script, database pipeline, or Google Sheets API integration, even if tabular data is involved.Stars: 196kUpdated Claude CodeCodexVerifiedInstall
shipping-and-launchaddyosmani/agent-skillsPrepares production launches. Use when preparing to deploy to production, or when asking what needs to be in place before shipping. Use when you need a pre-launch checklist, when setting up monitoring, when planning a staged rollout, or when you need a rollback strategy.Stars: 118kUpdated Claude CodeCodexVerifiedInstall
childhoodthedotmack/claude-memBabysit a shipped release after merge, publication, or deployment. Verify published packages, production behavior, installed plugins, and late CI or user reports until the observation window is clean; use babysit for PR readiness before release.Stars: 118kUpdated Claude CodeCodexVerifiedInstall