Skip to content

topprismdata/cultivating-ml-agent / skills/examples/cv-lb-gap-acknowledgment

cv-lb-gap-acknowledgment

Verified

CV-LB Gap Acknowledgment: CV improvement does NOT equal LB improvement. This is one of the most important MLOps principles. Use when: (1) OOF score keeps improving but LB score plateaus or drops, (2) Spending days tuning hyperparameters without LB improvement, (3) Comparing models on CV only without LB validation, (4) Trusting cross-validation as the "final" metric. Covers the mathematical reasons for CV-LB gap (overfitting to OOF noise, distribution shift train vs test, hyperparameter over-tuning to CV), empirical validation (0.005-0.01 gap observed in tabular competitions), the 5-stage validation pipeline (CV → submission → LB → analysis), when to stop iterating on CV alone, and the 80/20 rule for time allocation between CV optimization and LB validation.

CV-LB Gap Acknowledgment

Problem

A common MLOps trap: trusting CV score as the final metric.

Reality:

  • Many experiments show CV improvement but LB degradation
  • The gap can be 0.005-0.01 (5-10% of total error) on tabular competitions
  • Days of CV optimization may yield worse LB results
  • "Best CV" ≠ "Best LB"

The trap: Continue iterating on CV because "the score is improving", missing that CV is overfitting to noise.

Context / Trigger Conditions

Use this skill when:

  • OOF score keeps improving but LB score plateaus or drops (red flag!)
  • Spending days tuning hyperparameters without LB improvement
  • Comparing models on CV only without LB validation
  • Trusting cross-validation as the "final" metric for submissions
  • Adding more features because CV says so
  • More complex model beats simpler one on CV
  • Longer training (more iterations) improves CV

Solution: Mandatory 5-Stage Validation Pipeline

Stage 1: CV (Cross-Validation)
  ↓ Compute OOF score
Stage 2: Submission
  ↓ Submit to LB
Stage 3: LB Validation
  ↓ Compare CV vs LB
Stage 4: Analysis
  ↓ Diagnose gap
Stage 5: Decision
  ↓ Iterate / Pivot / Stop

Never skip Stage 2-4! CV is necessary but not sufficient.

Mathematical Reasons for CV-LB Gap

1. Overfitting to OOF Noise

OOF predictions are still predictions on training data (just held out in folds). They are not independent of training process.

Variance decomposition:

Var(OOF) = Var(true_error) + Var(model_selection) + Var(noise)
  • OOF score includes model selection variance (you picked this model because OOF was high)
  • LB score is out-of-sample (no selection bias)

2. Distribution Shift (Train vs Test)

  • Train: Same distribution you optimize on
  • Test: May have:
    • Different feature distributions (covariate shift)
    • Different label distributions (prior shift)
    • Different relationships (concept shift)

Example (Spaceship Titanic):

  • Train: 8693 rows
  • Test: 4277 rows (different distribution)
  • CV-LB gap: 0.005-0.01 (0.8124 OOF → 0.8078 LB)

3. Hyperparameter Over-tuning to CV

When you optimize hyperparameters based on CV score, you implicitly fit to CV noise.

Example:

  • Hyperparameter search finds: depth=7, lr=0.01, n_est=3000 (best on CV)
  • This specific config may be lucky on this CV split but not on test
  • True generalization: depth=6, lr=0.02 (more robust, slightly worse CV)

4. Submission Variance

Kaggle's public LB uses ~50% of test set. Private LB (final) uses 100%. Public LB ≠ Private LB in some competitions.

Empirical Validation (2026-06-13)

ExperimentOOFLBGapVerdict
House Prices V7 (XGBoost)0.1287850.13804+0.009CV 改善但 LB 也退化
House Prices V11 (Stack LR)0.1412981.16254+1.02❌ 完全失败
Spaceship V6 (3 new features)0.81360.80687-0.0067CV 高 LB 低
Spaceship V6 (Top-5)0.81360.80430-0.0093CV 高 LB 低
Spaceship V4 (15 CAT)0.81310.80570-0.0074CV 高 LB 低
Spaceship V2 (Top-5 CAT)0.81260.80780-0.0048Best LB (lowest gap)
Spaceship V8 (AutoGluon)0.8317pendingTBDTBD

Key Observations:

  • Gap is consistent: 0.005-0.01 across experiments
  • More complex features → bigger gap (V6 vs V2)
  • Simpler is better: V2 (basic 5 CatBoost) had lowest CV-LB gap

4 Practical Rules

Rule 1: Submit After Major CV Improvements

# After each major CV improvement, submit to LB
if oof_improved_by_0.001:
    submit_to_lb()
    compare_cv_vs_lb()

Don't wait for "perfect" CV. Submit early and often.

Rule 2: Trust the Simpler Model When CV-LB Gap is Large

If model A has CV 0.001 better than model B but LB 0.005 worse:

  • Choose model B (simpler, more robust)
  • CV improvement of 0.001 may be noise
  • LB degradation of 0.005 is real

Rule 3: Allocate 80% Time to LB Validation, 20% to CV Optimization

time_allocation = {
    'cv_optimization': 0.20,  # 20% on improving CV
    'lb_validation': 0.80,     # 80% on LB validation + analysis
}

Why?: CV is faster but unreliable. LB is slower but real.

Rule 4: Watch for the "CV Improvement Trap"

# Red flag pattern:
for v in [v1, v2, v3, ...]:
    if v.oof > previous.oof:
        print(f"V{v} CV improved! Continue iterating")
        submit_to_lb()
    else:
        print(f"V{v} CV worse. Pivot to new approach")

Better:

for v in [v1, v2, v3, ...]:
    if v.oof > previous.oof and v.lb > previous.lb:
        print(f"V{v} Both CV and LB improved. Continue")
    elif v.oof > previous.oof and v.lb < previous.lb:
        print(f"V{v} CV up but LB down. STOP, pivot")
    else:
        print(f"V{v} Both worse. Pivot")

When to STOP Iterating on CV

Stop if any of these are true:

  • 3 consecutive CV improvements all have LB degradation
  • CV is improving but LB is plateauing (within noise of best LB)
  • You can't improve CV by >0.001 (noise level)
  • Time budget exhausted (80% of time on CV, 20% on LB)

How to Diagnose CV-LB Gap

Step 1: Submit Current Best CV Model

best_cv_model = max(experiments, key=lambda e: e.oof)
best_cv_model.submit_to_lb()

Step 2: Analyze

gap = best_cv_model.lb - best_cv_model.oof
if gap > 0.01:  # Large gap
    print("CV-LB gap > 0.01: Model is overfitting CV")
    print("Try: simpler model, more regularization, fewer features")
elif gap < 0.005:  # Small gap
    print("CV-LB gap < 0.005: Model is generalizing well")
    print("Continue iterating on CV")
else:  # Medium gap
    print("CV-LB gap 0.005-0.01: Moderate overfitting")
    print("Submit and validate before continuing")

Step 3: Compare to Baseline

baseline_gap = baseline_model.lb - baseline_model.oof
if current_gap > 2 * baseline_gap:
    print("Gap is 2x worse than baseline. Revert to baseline")

Empirical Pattern: Which CV Improvements Transfer to LB?

Transfer well:

  • Simple model improvements (fewer features, more regularization)
  • Better preprocessing (missing values, encoding)
  • Ensemble of diverse algorithms (AutoGluon, multi-family GBDT)

Transfer poorly:

  • Adding many engineered features (especially interactions)
  • Aggressive hyperparameter tuning
  • More model variants (15+ same family)
  • Target encoding without nested CV
  • Log-transform / Box-Cox on top of base features

Anti-Patterns to Avoid

❌ Don't trust CV alone (always validate on LB) ❌ Don't continue iterating when CV improves but LB doesn't ❌ Don't add complexity without LB validation ❌ Don't tune hyperparameters extensively without LB check ❌ Don't ignore the CV-LB gap (it's a signal, not noise)

Practical Workflow

# Stage 1: Quick CV experiment
v1 = Experiment()
v1.run_cv()
v1.submit_to_lb()  # Always submit after first CV

# Stage 2: Iterate
for v in [v2, v3, v4]:
    v.run_cv()
    if v.oof > previous.oof:
        v.submit_to_lb()  # Submit if CV improved
        if v.lb < previous.lb:
            print("CV up but LB down. Revert to previous version")
            v = previous  # Revert

# Stage 3: Final
best_lb_version = max(submitted_versions, key=lambda v: v.lb)
best_lb_version.save_submission()

Related Skills

  • ml-sweet-spot — When to stop adding complexity
  • autogluon-first — AutoGluon is robust to CV-LB gap
  • multi-model-diversity — Diversity reduces overfitting to CV
  • controlled-submission-experiment — How to test changes safely
  • progressive-verification-debugging — Validate each step

Validation Source

  • House Prices: 0.009 gap (V7), 1.02 gap (V11/V12 catastrophic)
  • Spaceship Titanic: 0.005-0.01 gap across all V2-V8 experiments
  • Pattern observed across 2+ competitions: gap is real and consistent
finishing-a-development-branchobra/superpowersUse when implementation is complete, all tests pass, and you need to decide how to integrate the workStars: 322kUpdated Claude CodeCodexVerifiedInstall
systematic-debuggingobra/superpowersUse when encountering any bug, test failure, or unexpected behavior, before proposing fixesStars: 322kUpdated Claude CodeCodexVerifiedInstall
test-driven-developmentobra/superpowersUse when implementing any feature or bugfix, before writing implementation codeStars: 322kUpdated Claude CodeCodexVerifiedInstall
skill-creatoranthropics/skillsCreate new skills, modify and improve existing skills, and measure skill performance. Use when users want to create a skill from scratch, edit, or optimize an existing skill, run evals to test a skill, benchmark skill performance with variance analysis, or optimize a skill's description for better triggering accuracy.Stars: 197kUpdated Claude CodeCodexVerifiedInstall