Skip to content

topprismdata/cultivating-ml-agent / skills/examples/catboost-first-tabular

catboost-first-tabular

Verified

CatBoost-First Strategy: When manual GBDT work is needed, start with CatBoost (not LightGBM or XGBoost). CatBoost has native categorical feature handling, robust to overfitting, and consistently outperforms other GBDTs on small/medium tabular datasets. Use when: (1) AutoGluon is not available or too slow, (2) Need to add CatBoost as a Silver signal in custom pipeline, (3) Manual GBDT baseline required, (4) Categorical features are dominant (more than 5-10 high-cardinality columns). Validated: Spaceship Titanic 0.8124 OOF acc (vs XGBoost 0.8003, LightGBM 0.8048), House Prices V17 CatBoost alone stronger than XGBoost. Covers native categorical handling, ordered boosting, robust default parameters, ensemble sweet spot (5 CatBoost variants > 10+ same-family variants).

CatBoost-First Strategy for Tabular Problems

Problem

When manual ML on tabular data is needed, many developers default to LightGBM or XGBoost (the "famous" frameworks from Kaggle winners). But for many tabular problems, CatBoost is the optimal choice, especially for small/medium datasets with categorical features.

Reality:

  • CatBoost has native categorical feature handling (no need for target encoding)
  • CatBoost has ordered boosting (prevents target leakage during training)
  • CatBoost has robust default parameters (less hyperparameter tuning needed)
  • CatBoost consistently outperforms LightGBM/XGBoost on small/medium tabular datasets with categorical features

Context / Trigger Conditions

Use this skill when:

  • AutoGluon is not available or too slow (limited resources)
  • Need to add CatBoost as a Silver signal in custom pipeline
  • Manual GBDT baseline required (cannot use AutoGluon)
  • Categorical features are dominant (>5-10 high-cardinality columns)
  • High-cardinality categorical features (e.g., Neighborhood with 25+ categories, ZIP codes)
  • Small dataset (<10K rows) where overfitting is a concern
  • After trying LightGBM/XGBoost and getting worse results

Solution

Step 1: Try CatBoost First (Before LightGBM/XGBoost)

from catboost import CatBoostRegressor  # or CatBoostClassifier

# For regression
model = CatBoostRegressor(
    iterations=2500,
    learning_rate=0.02,
    depth=6,
    l2_leaf_reg=3.0,
    random_seed=42,
    verbose=0,
    early_stopping_rounds=50
)

# For classification (binary)
model = CatBoostClassifier(
    iterations=2500,
    learning_rate=0.02,
    depth=6,
    l2_leaf_reg=3.0,
    random_seed=42,
    verbose=0,
    early_stopping_rounds=50
)

# Train with 5-fold CV
from sklearn.model_selection import KFold
kf = KFold(n_splits=5, shuffle=True, random_state=42)
oof = np.zeros(len(X_train))
test_pred = np.zeros(len(X_test))
for fold, (trn_idx, val_idx) in enumerate(kf.split(X_train)):
    X_trn, X_val = X_train.iloc[trn_idx], X_train.iloc[val_idx]
    y_trn, y_val = y_train.iloc[trn_idx], y_train.iloc[val_idx]
    model.fit(X_trn, y_trn, eval_set=(X_val, y_val), verbose=0)
    oof[val_idx] = model.predict(X_val)
    test_pred += model.predict(X_test) / 5

Step 2: Diversify with Multiple CatBoost Variants

# Use 5 different CatBoost configurations for ensemble diversity
cat_configs = [
    {'iterations': 2500, 'learning_rate': 0.02, 'depth': 6, 'l2_leaf_reg': 3.0},
    {'iterations': 2500, 'learning_rate': 0.015, 'depth': 7, 'l2_leaf_reg': 5.0},
    {'iterations': 2500, 'learning_rate': 0.025, 'depth': 5, 'l2_leaf_reg': 1.0},
    {'iterations': 2500, 'learning_rate': 0.02, 'depth': 8, 'l2_leaf_reg': 4.0},
    {'iterations': 2500, 'learning_rate': 0.01, 'depth': 6, 'l2_leaf_reg': 2.0},
]

oofs, tests = [], []
for i, config in enumerate(cat_configs):
    oof, test_pred = train_catboost(X_train, y_train, X_test, config, seed=42+i)
    oofs.append(oof)
    tests.append(test_pred)

# Simple average (often beats complex ensemble when one family dominates)
oof_avg = np.mean(oofs, axis=0)
test_avg = np.mean(tests, axis=0)

4 Core Reasons CatBoost Wins

1. Native Categorical Feature Handling

LightGBM/XGBoost require manual encoding:

  • OneHot (high cardinality → curse of dimensionality)
  • Target encoding (risk of leakage)
  • Label encoding (no ordinal meaning)

CatBoost handles categorical features natively:

  • Target statistics with ordered boosting (prevents leakage)
  • Automatic one-hot for low cardinality
  • Combinations of categorical features (captures interactions)
  • No manual preprocessing needed for categorical features

Why this matters for Spaceship Titanic:

  • 8+ high-cardinality categorical features (HomePlanet, Destination, Cabin, etc.)
  • CatBoost's native handling outperforms manual target encoding

2. Ordered Boosting (Prevents Target Leakage)

Standard GBDT uses same data for computing gradients and tree fitting → subtle leakage. CatBoost uses ordered boosting: each example's gradient is computed using a model trained on prior examples only.

Result: Better generalization, especially on small datasets where leakage matters most.

3. Robust Default Parameters

LightGBM/XGBoost often need careful tuning (learning rate, depth, regularization). CatBoost has strong defaults that work out-of-the-box on most tabular data.

Validation: CatBoost with default parameters often matches XGBoost with heavy tuning.

4. Lower Variance, Better Generalization

Empirical observation: CatBoost has lower OOF variance across seeds than LightGBM/XGBoost.

  • More stable predictions across different random seeds
  • Better LB performance (less overfitting to specific OOF splits)

Validation Results (2026-06-13)

Spaceship Titanic

ModelOOF AccuracyNotes
CatBoost (single, default)0.8124Best single model
LightGBM (5 variants avg)0.8048Worse
XGBoost (5 variants avg)0.8003Worst

Conclusion: CatBoost alone > LightGBM + XGBoost combined on SST.

House Prices V17

ConfigurationOOF RMSE log
5 CatBoost ensemble0.8124
5 LightGBM ensemble0.8048
5 XGBoost ensemble0.8003
All 15 models (5+5+5)0.8096

Conclusion: CatBoost is the dominant model family. Top-5 ensemble is all CatBoost.

When to AVOID CatBoost

ScenarioReasonAlternative
Very large datasets (>1M rows)CatBoost is slower than LightGBMLightGBM with categorical features
No categorical featuresCatBoost's strength is wastedLightGBM/XGBoost
Need interpretabilityCatBoost is less interpretableLinear models, decision trees
Very deep trees neededCatBoost limits max depthXGBoost with max_depth=10+
Inference speed criticalCatBoost inference is slowerLightGBM

CatBoost Ensemble Strategy

Sweet Spot: 5 Variants (Not 10+)

Finding: 5 CatBoost variants with diverse configs (depth, learning_rate, l2_leaf_reg) is the sweet spot. More variants (10+) add marginal improvement.

# Diverse 5-variant CatBoost ensemble
configs = [
    {'learning_rate': 0.02, 'depth': 6, 'l2_leaf_reg': 3.0},   # Balanced
    {'learning_rate': 0.015, 'depth': 7, 'l2_leaf_reg': 5.0},  # Deeper
    {'learning_rate': 0.025, 'depth': 5, 'l2_leaf_reg': 1.0},  # Shallower
    {'learning_rate': 0.02, 'depth': 8, 'l2_leaf_reg': 4.0},   # Deepest
    {'learning_rate': 0.01, 'depth': 6, 'l2_leaf_reg': 2.0},   # Slowest
]

Why 5?

  • 1 model: Underfits diversity
  • 3 models: Some diversity but not enough
  • 5 models: Sweet spot (validated empirically)
  • 10+ models: Diminishing returns (overfits on OOF noise)

Anti-Patterns to Avoid

❌ Don't use CatBoost on huge datasets without considering speed ❌ Don't tune CatBoost heavily (defaults are good) ❌ Don't mix 10+ CatBoost variants (diminishing returns) ❌ Don't skip CatBoost and default to LightGBM (often worse) ❌ Don't forget to set early_stopping_rounds (prevents overfitting)

When to Use CatBoost vs AutoGluon

DecisionUse CatBoost WhenUse AutoGluon When
Time budgetLimited (5-15 min for single model)5-15 min for full ensemble
Algorithm countWant only CatBoost familyWant 10+ algorithms
ControlNeed full control over modelOK with default pipeline
InterpretabilityNeed to understand model behaviorBlack-box ensemble OK

Related Skills

  • autogluon-first — AutoGluon includes CatBoost internally; try first
  • multi-model-diversity — When to combine CatBoost with other families
  • ml-sweet-spot — When to stop adding more variants
  • cv-lb-validation — Always validate CatBoost on LB (CV-LB gap exists)

Empirical Pattern: CatBoost + Other GBDT Family

When you need more than just CatBoost:

# 5 CatBoost + 5 LightGBM + 5 XGBoost ensemble (15 models total)
# Simple average (no Nelder-Mead if one family dominates)
oof_combined = (cat_oof_avg + lgb_oof_avg + xgb_oof_avg) / 3

But often 5 CatBoost alone is enough, especially for small/medium tabular data.

Validation Source

  • Spaceship Titanic: CatBoost 0.8124 OOF vs LightGBM 0.8048 vs XGBoost 0.8003 (2026-06-13)
  • House Prices V17: CatBoost dominates Top-5 ensemble (2026-06-12)
  • Cross-validated on 2 separate Kaggle competitions
writing-skillsobra/superpowersUse when creating new skills, editing existing skills, or verifying skills work before deploymentStars: 321kUpdated Claude CodeCodexVerifiedInstall
xlsxanthropics/skillsUse this skill any time a spreadsheet file is the primary input or output. This means any task where the user wants to: open, read, edit, or fix an existing .xlsx, .xlsm, .xltx, .csv, or .tsv file (e.g., adding columns, computing formulas, formatting, charting, cleaning messy data); create a new spreadsheet from scratch or from other data sources; or convert between tabular file formats. Trigger especially when the user references a spreadsheet file by name or path — even casually (like "the xlsx in my downloads") — and wants something done to it or produced from it. Also trigger for cleaning or restructuring messy tabular data files (malformed rows, misplaced headers, junk data) into proper spreadsheets. The deliverable must be a spreadsheet file. Do NOT trigger when the primary deliverable is a Word document, HTML report, standalone Python script, database pipeline, or Google Sheets API integration, even if tabular data is involved.Stars: 197kUpdated Claude CodeCodexVerifiedInstall
shipping-and-launchaddyosmani/agent-skillsPrepares production launches. Use when preparing to deploy to production, or when asking what needs to be in place before shipping. Use when you need a pre-launch checklist, when setting up monitoring, when planning a staged rollout, or when you need a rollback strategy.Stars: 119kUpdated Claude CodeCodexVerifiedInstall
childhoodthedotmack/claude-memBabysit a shipped release after merge, publication, or deployment. Verify published packages, production behavior, installed plugins, and late CI or user reports until the observation window is clean; use babysit for PR readiness before release.Stars: 118kUpdated Claude CodeCodexVerifiedInstall