Skip to content

topprismdata/cultivating-ml-agent / skills/examples/autogluon-preset-strategy

autogluon-preset-strategy

Verified

AutoGluon preset selection strategy: when to use medium/good/high/best_quality, when EDA is unnecessary, and how to tune within a preset. Validated on s6e7 (AG high_quality 600s → OOF=0.8739, LB=0.87458, gap=0.0007 — perfect alignment). Use when: (1) Deciding which AutoGluon preset to start with, (2) Wondering whether to do manual EDA before fitting, (3) Wanting to know which hyperparameters to tune first, (4) Deciding time budget for tabular training. Differs from `autogluon-first` (which says "just use best_quality"): this skill explains the preset tradeoffs and where manual intervention pays off vs not.

AutoGluon Preset Strategy

Problem

autogluon-first says "use best_quality preset always". But:

  • best_quality takes 1-4 hours — wasted if you only need a quick baseline
  • medium_quality is the default but doesn't enable bagging/stacking → no OOF predictions
  • The choice between presets has a 5x time × accuracy tradeoff that the existing skill doesn't quantify

Decision Tree

N = training rows
N < 10K   → presets='best_quality' (single GBDT plateau is real concern)
N 10K-100K → presets='good_quality' first (10-30 min), upgrade to high if time allows
N > 100K  → presets='good_quality' is enough (data variance dominates model choice)

GPU available? 
  No → presets='high_quality' max (TabPFNv2 needs GPU)
  Yes → consider 'extreme_quality' (uses TabPFNv2, TabICL, Mitra)

Metric?
  balanced_accuracy / F1 → add calibrate_decision_threshold=True
  accuracy → default ('auto')
  log_loss → use predict_proba, no calibration

Preset Reference (from AG 1.4.0 docstring)

PresetTrain TimeOOF Available?ModelsWhen
medium_quality (default)~5min❌ No (auto_stack=False)3NEVER for serious use
good_quality10-30min✅ Yes12Default starting point
high_quality30-60min✅ Yes14When good isn't enough
best_quality1-4h✅ Yes~100Kaggle competition
extreme_quality2-8h✅ Yes22+TabFMGPU only, 30K rows ideal

Why medium_quality has no OOF: AG's predict_oof() requires bag mode (auto_stack=False means no bag). The "val score" reported is in-sample on the holdout — not real OOF. Don't trust it.

Empirical Validation (s6e7, N=690K, 3-class)

PresetTimeModelsBest ValOOFOOF/Val gap
medium_quality182s30.8825N/AN/A
good_quality376s120.87300.87300
high_quality947s140.87390.87390

Surprising finding: medium_quality's val score (0.8825) is higher than good/high (0.873). This is fake: medium has no bag/stack, val is in-sample on the internal holdout. The true OOF (good=0.8730, high=0.8739) is what you should trust.

Practical implication: good→high gives only +0.0009 OOF improvement at 2.5x time cost. good_quality is the sweet spot unless you have hours to burn.

The "Do I Need EDA?" Question

Short answer: For standard tabular (numeric + categorical features, no time series, no text), you do NOT need EDA before fitting AG.

Data TypeEDA Before AG?Why
Numeric + categorical (standard)❌ NoAG handles missing/encoding automatically
Heavy missingness (>30%)❌ NoAG's NN_TORCH handles it natively
Class imbalance (>10:1)❌ NoSet eval_metric='balanced_accuracy'
Time series✅ YesAG default doesn't create lag/rolling features
Text features⚠️ OptionalAG has NGRAM auto-generation; manual helps sometimes
Image features✅ YesNeed AG_AUTOMM, different setup
Business rules✅ YesEncode rules into target/feature

Validation: s6e7 (690K rows, 14 features, ~50% missing) — fit AG directly with no manual EDA. Result: OOF=0.8739, LB=0.87458, gap=0.0007. No EDA was needed.

Tuning Within a Preset

After baseline (good_quality, 30 min), try in order:

Priority 1: Increase time_limit (cheapest)

predictor.fit(tr, time_limit=3600, presets='good_quality')  # 6x longer

Priority 2: Customize key models

predictor.fit(tr, presets='good_quality',
              hyperparameters={'GBM': {'num_leaves': 64, 'learning_rate': 0.03},
                              'CAT': {'depth': 8}})

Priority 3: Use refit_full (more data = better)

predictor.fit(tr, presets='high_quality', refit_full=True)
# refit trains on ALL data (no holdout) — typically +0.001-0.005 OOF

Priority 4: Decision threshold calibration

predictor.fit(tr, eval_metric='balanced_accuracy',
              calibrate_decision_threshold=True)
# Helps when metric is balanced_accuracy or F1

Standard Workflow

# Step 1: Baseline (10-30 min)
predictor = TabularPredictor(label='target', eval_metric='accuracy').fit(
    tr, time_limit=600, presets='good_quality')

# Step 2: Inspect (1 min)
print(predictor.leaderboard(silent=True))
print(predictor.feature_importance(tr))

# Step 3: If you need more (decide based on Step 2)
# - Models are similar → refit_full=True
# - One model dominates → tune hyperparameters
# - Need more time → longer time_limit
predictor = TabularPredictor(label='target', eval_metric='accuracy').fit(
    tr, time_limit=3600, presets='good_quality',
    hyperparameters={'GBM': {'num_leaves': 64}})

# Step 4: Predict and submit
pred = predictor.predict(te)

When AG Wins and Loses

AG Wins:

  • Standard tabular with 1K-1M rows
  • No time budget constraints
  • You don't need to understand model decisions
  • Multiple data types (numeric + categorical + text) — AG handles all

AG Loses:

  • Real-time inference (<10ms latency) — AG ensembles are slow
  • Custom loss functions
  • Tiny data (<100 rows) — GBDT overfits, AG can't help
  • Adversarial / non-stationary targets — AG assumes IID
  • You need SHAP/explainability for each model decision

Anti-Patterns

Anti-patternWhy wrongFix
Use medium_quality because it's defaultNo bag/stack, no OOFUse good_quality
Skip AG and start manual LightGBMHours vs minutesRun AG baseline first
Spend days tuning GBDT hyperparametersAG already HPO-tunesTrust AG's defaults
Do extensive EDA before fitting AGAG auto-handles standard casesJust fit, inspect, then iterate
Hand-craft 30+ features when AG already doesDiminishing returnsAdd only domain-specific signals

Key API Methods

# Inspect
predictor.leaderboard(silent=True)              # Model rankings + val scores
predictor.feature_importance(tr)                 # Permutation importance
predictor.model_names()                         # List of trained models
predictor.info()                                # Full info

# Predict
predictor.predict(te)                           # Class labels
predictor.predict_proba(te)                     # Class probabilities

# Inspect best model
best_model = predictor.model_best
predictor.predict_oof()                          # Only if bag mode enabled
predictor.refit_full()                          # Retrain best on all data

When to Use This Skill vs autogluon-first

ScenarioUse this skillUse autogluon-first
Choose first preset✅❌ (says use best_quality always)
Decide if EDA needed✅❌
Optimize within preset✅❌
Strategic "why use AG" reasoning❌✅
Quick reference of AG principles❌✅

References

  • AG 1.4.0 docstring (inspect.getsource on TabularPredictor.fit)
  • AutoGluon 1.4.0 release notes
  • Erickson et al. (2020) "AutoGluon-Tabular" arXiv:2003.06505
  • Salinas & Erickson (2025) "TabRepo" arXiv:2507.07829 (zeroshot hyperparameter portfolio)
reverse-engineer-anythingmorluto/reaReverse engineer native, managed, Electron/JavaScript, packaged, firmware, and browser targets with REA, and build agent environments (AgentEnv bundles) of web applications from a live instance or a capture, and of Linux desktop apps with a task from a request. Use shipped-artifact or requested runtime evidence to explain features, compare versions, decompile code, or guide a reconstruction. Skip REA for ordinary source-repository architecture analysis.Stars: 471kUpdated Claude CodeCodexVerifiedInstall
diagnosing-superpowersobra/superpowersUse when a superpowers session went wrong and your human partner wants to know why — repeated work, ignored plans, stumbles, poor results, a skill that didn't fire, "it took too long", "why is it so expensive", "what is it doing" — or wants to build a bug report for the superpowers maintainers, for the current session or a past one identified by id or path, on any harness.Stars: 322kUpdated Claude CodeCodexVerifiedInstall
dispatching-parallel-agentsobra/superpowersUse when facing 2+ independent tasks that can be worked on without shared state or sequential dependenciesStars: 322kUpdated Claude CodeCodexVerifiedInstall
executing-plansobra/superpowersUse when executing an implementation plan in the current session as the implementer yourself — your human partner chose inline execution, or no subagent tool is availableStars: 322kUpdated Claude CodeCodexVerifiedInstall