topprismdata/cultivating-ml-agent / skills/examples/multi-level-aggregation-overfitting
multi-level-aggregation-overfitting
VerifiedUse when: (1) considering adding family-level or store-level aggregation features to day-specific models, (2) tempted to copy hierarchical features from 1st place solutions, (3) CV improves but LB degrades after adding aggregated features. WARN: Multi-level aggregation features that work in unified models may OVERFIT in day-specific (direct) forecasting frameworks.
Multi-Level Aggregation Overfitting in Day-Specific Models
Problem
1st place solutions often use family-level and store-level aggregation features (family lag, store rolling mean, store-family ratio features). However, when applied to day-specific models (separate model per horizon day), these features can cause overfitting: CV improves but LB degrades.
Why It Overfits
In day-specific models, each model trains on ~2.87M samples covering all (store, family) pairs. The model already sees sufficient examples per pair. Family/store level features add redundancy:
-
Leakage via aggregation: Family-level lags are highly correlated with individual store-family lags (especially for families with few stores). The ratio features (
sf_to_fam_ratio) may capture noise rather than signal. -
Cross-pair interference: Day-specific models learn patterns across all pairs simultaneously. Adding aggregated features increases the feature space without adding truly independent information.
-
CV overfitting: The expanding-window CV may not penalize these features enough because the family/store patterns are stable across time, but they don't generalize to the test period's specific dynamics.
Evidence
| Version | Features | CV RMSLE | LB |
|---|---|---|---|
| R11b (base) | 82 features | 0.42041 | 0.40073 |
| R12 multilevel | 82 + 18 agg | 0.41668 (better) | 0.39874 (worse vs R11c=0.39824) |
CV improved by 0.00373 but LB got worse by 0.00050 relative to R11c.
In contrast, YoY features (only 4 features) improved LB by 0.00045 with less CV improvement.
Rule of Thumb
- Unified model: Multi-level aggregation likely helps (model needs hints about hierarchy)
- Day-specific model: Multi-level aggregation may overfit (model already sees all pairs)
- Safer alternative: Use target encoding at family/store level instead of raw aggregation
- Feature budget: Prefer fewer high-signal features (like YoY) over many correlated ones
Related
yoy-364day-features— the feature that actually worked insteadcontrolled-submission-experiment— how we isolated this findingts-day-specific-forecasting— the framework where this applies
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| slack-gif-creatoranthropics/skillsKnowledge and utilities for creating animated GIFs optimized for Slack. Provides constraints, validation tools, and animation concepts. Use when users request animated GIFs for Slack like "make me a GIF of X doing Y for Slack." | Stars: 198k | Updated | Claude CodeCodex | Verified | Install |
| ponytailDietrichGebert/ponytailLazy senior dev mode for any coding task: the smallest change that fully solves it, and a reply a busy human understands in one read. | Stars: 194k | Updated | Claude CodeCodex | Verified | Install |
| ponytail-debtDietrichGebert/ponytailHarvest every shortcut: comment into one debt ledger, so deferrals get tracked instead of forgotten. One-shot report. | Stars: 194k | Updated | Claude CodeCodex | Verified | Install |
| ponytail-debtDietrichGebert/ponytailList every `shortcut:` comment (and older `ponytail:` ones) as a debt ledger. One-shot report, changes nothing. Use for "ponytail debt", "what did ponytail defer", "list the shortcuts", /ponytail-debt. | Stars: 194k | Updated | Claude CodeCodex | Verified | Install |