sb-dev/openquant / .agents/skills/ml4t-lookahead-bias
ml4t-lookahead-bias
VerifiedDetect future information leaking into features, labels, or evaluation. Use when any pipeline step might expose data not yet available at prediction time.
Lookahead Bias
The most common ML4T failure. A model uses future information during training - normalization with full-sample statistics, labels from future thresholds, standard k-fold CV on time series - producing a great backtest that fails immediately in production.
The Problem
Lookahead bias means using information that would not have been available at prediction time. It inflates backtest Sharpe and is the #1 reason strategies fail live. Common sources: full-sample normalization, future-based thresholds, train+test preprocessing, and shuffled CV on time series.
The Pattern
WRONG
from sklearn.preprocessing import StandardScaler
# Normalize features using full dataset statistics (future leak)
scaler = StandardScaler()
X_scaled = scaler.fit_transform(X) # mean/std computed on ALL data including test
# Label threshold from full history (future leak)
threshold = returns.quantile(0.75) # Uses future returns to set threshold
y = (forward_returns > threshold).astype(int)
# Shuffled k-fold on time series (future leak)
from sklearn.model_selection import KFold
cv = KFold(n_splits=5, shuffle=True) # Training on 2024 data, testing on 2020
CORRECT
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import Pipeline
# Expanding-window normalization: only past data
X_normalized = (X - X.expanding().mean().shift(1)) / X.expanding().std().shift(1)
# Expanding threshold: only past returns
threshold = returns.expanding().quantile(0.75).shift(1)
y = (forward_returns > threshold).astype(int)
# Time-series CV that preserves order
from sklearn.model_selection import TimeSeriesSplit
cv = TimeSeriesSplit(n_splits=5)
Preprocessing Pipeline
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import Ridge
# Pipeline ensures scaler is fit on train only, even inside CV
pipe = Pipeline([
("scaler", StandardScaler()),
("model", Ridge()),
])
pipe.fit(X_train, y_train) # scaler sees only training data
pipe.predict(X_test) # test data transformed with train stats
Detection
- In-sample and out-of-sample performance are suspiciously similar (gap < 5%)
- Sharpe ratio > 2.0 on daily data with a simple model
- Feature values change when you recompute with more recent data appended
- Performance degrades sharply when switching from k-fold to walk-forward CV
- Labels use close-to-close returns but execution is next-open - this 50-100 bps gap per trade is hidden lookahead
Guardrails
- All rolling/expanding statistics must use
.shift(1)to exclude the current observation fit_transform()must never touch test data - usePipelineor manual train/test splits- Cross-validation must respect temporal order - no
shuffle=Trueon time series - Label thresholds must be computed from past data only (expanding window)
- Point-in-time fundamentals: use report dates, not period dates
Production Implementation
from ml4t.diagnostic.splitters import CombinatorialCV
from ml4t.engineer import create_dataset_builder
builder = create_dataset_builder(features, labels, dates=timestamps, scaler="standard")
cv = CombinatorialCV(n_groups=8, n_test_groups=2, embargo_pct=0.01)
fold = next(builder.split(cv))
X_train, y_train = fold.X_train, fold.y_train
X_test, y_test = fold.X_test, fold.y_test
Checklist
- All normalizations use expanding window with
.shift(1), not full-sample stats -
fit_transform()only on training data (usePipelineor explicit split) - Time-series CV with purging and embargo (not shuffled k-fold)
- Labels do not use future thresholds or statistics
- Point-in-time data: report dates used, not period-end dates
- Suspicious results investigated: Sharpe > 2, IS/OOS gap < 5%
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| writing-skillsobra/superpowersUse when creating new skills, editing existing skills, or verifying skills work before deployment | Stars: 319k | Updated | Claude CodeCodex | Verified | Install |
| xlsxanthropics/skillsUse this skill any time a spreadsheet file is the primary input or output. This means any task where the user wants to: open, read, edit, or fix an existing .xlsx, .xlsm, .xltx, .csv, or .tsv file (e.g., adding columns, computing formulas, formatting, charting, cleaning messy data); create a new spreadsheet from scratch or from other data sources; or convert between tabular file formats. Trigger especially when the user references a spreadsheet file by name or path — even casually (like "the xlsx in my downloads") — and wants something done to it or produced from it. Also trigger for cleaning or restructuring messy tabular data files (malformed rows, misplaced headers, junk data) into proper spreadsheets. The deliverable must be a spreadsheet file. Do NOT trigger when the primary deliverable is a Word document, HTML report, standalone Python script, database pipeline, or Google Sheets API integration, even if tabular data is involved. | Stars: 196k | Updated | Claude CodeCodex | Verified | Install |
| shipping-and-launchaddyosmani/agent-skillsPrepares production launches. Use when preparing to deploy to production, or when asking what needs to be in place before shipping. Use when you need a pre-launch checklist, when setting up monitoring, when planning a staged rollout, or when you need a rollback strategy. | Stars: 118k | Updated | Claude CodeCodex | Verified | Install |
| childhoodthedotmack/claude-memBabysit a shipped release after merge, publication, or deployment. Verify published packages, production behavior, installed plugins, and late CI or user reports until the observation window is clean; use babysit for PR readiness before release. | Stars: 118k | Updated | Claude CodeCodex | Verified | Install |