Skip to content

topprismdata/cultivating-ml-agent / skills/examples/kaggle-submission-format-by-metric

kaggle-submission-format-by-metric

Verified

Match the submission file format to the competition's evaluation metric BEFORE submitting. For ranking-based metrics (AUC, log_loss, MAP, NDCG, RMSLE) you MUST submit continuous probability/score values, not 0/1 class labels or rounded integers. Use when: (1) Preparing the final submission for any Kaggle competition, (2) AutoGluon's default `predict()` returns class labels for classification (must use `predict_proba()`), (3) Unsure whether to threshold predictions, (4) CV score is excellent but LB score is near-random. Real failure case: S6E2 Heart Disease rerun 2026-06-14 — OOF AUC 0.95554 dropped to Public LB 0.88403 with 0/1 submission, recovered to 0.95357 with probability submission.

Match Submission Format to Evaluation Metric

Problem

A perfect model can score near-random on the public leaderboard if the submission file format doesn't match the metric's expectations. The most common mistake: submitting 0/1 thresholded labels for a metric that needs continuous probabilities.

Real failure case (2026-06-14, S6E2 Heart Disease):

  • Model: AutoGluon best_quality ensemble, OOF AUC 0.95554
  • Submission A: submission_autogluon.csv (0/1 thresholded via predictor.predict())
    • Public LB: 0.88403 ❌ (looked like complete model failure)
    • Private LB: 0.88643
  • Submission B: submission_autogluon_proba.csv (continuous via predictor.predict_proba())
    • Public LB: 0.95357 ✅
    • Private LB: 0.95510
  • Same model, same OOF — only the submission format differed. 0.07 LB drop from thresholding alone.

Context / Trigger Conditions

Use this skill when:

  • About to submit the final (or any) prediction to a Kaggle competition
  • AutoGluon / sklearn / xgboost default predict() returns hard labels for classification
  • Competition metric is ranking-based (see list below)
  • CV score is great but LB score is suspiciously low
  • You see sample_submission.csv with 0.0/1.0 values (those are the target format, NOT necessarily the submission format)

DO NOT threshold for ranking-based metrics:

  • roc_auc, auc — area under ROC; needs continuous scores
  • auc_mu — multi-class AUC; needs full probability matrix
  • log_loss — penalizes confident wrong answers; needs probabilities
  • MAP, NDCG — ranking metrics; need scores
  • brier_score — squared error on probabilities
  • mean_columnwise_auc — column-wise AUC
  • rmse on log-target (RMSLE) — for log-transformed regression, submit log predictions directly

DO threshold (or round) for these metrics:

  • accuracy — predicted class label
  • f1, precision, recall — predicted class label
  • quadratic_kappa — rounded integer (for ordinal)
  • mae, rmse on raw target — continuous regression predictions are fine, but rounding is harmless

Solution: Metric → Format Decision Tree

Is metric ranking-based (AUC, log_loss, MAP, NDCG, etc.)?
├── YES → Submit continuous probabilities / scores
│         AutoGluon: predictor.predict_proba(test)
│         sklearn:   model.predict_proba(test)
│         xgboost:   model.predict_proba(test)
│
└── NO (accuracy, f1, kappa, etc.)
    └── Submit class labels (or rounded regression values)
        AutoGluon: predictor.predict(test)
        sklearn:   model.predict(test)

Quick reference per framework

AutoGluon:

# WRONG for AUC:
preds = predictor.predict(test)            # hard labels
# CORRECT for AUC:
preds = predictor.predict_proba(test)     # probability DataFrame
# For multi-class AUC, submit the full matrix; column order matters
# and must match sample_submission.csv column order.

scikit-learn:

# WRONG for AUC:
preds = model.predict(test)                # hard labels
# CORRECT for AUC:
preds = model.predict_proba(test)[:, 1]    # positive class probability

XGBoost / LightGBM / CatBoost:

# For binary AUC:
preds = model.predict_proba(test)[:, 1]
# For multi-class AUC (sklearn API):
preds = model.predict_proba(test)          # full matrix
# For multi-class AUC (learning API):
preds = model.predict(test)                # this IS probabilities in raw API

Check sample_submission.csv carefully

head -3 sample_submission.csv
  • Values are all 0 and 1 → metric is accuracy/f1 etc. → submit class labels
  • Values are 0.0 and 1.0 with target column → likely a target template (binary outcome), but metric is probably AUC. Submit probabilities anyway.
  • Values are floats in [0, 1] → metric is log_loss/probability calibration → submit probabilities
  • Values are integers in [0, 9] → quadratic_kappa/ordinal → submit integers
  • Values are floats with no obvious range → regression, submit raw predictions
  • Values are strings like Low/Medium/High → multi-class classification with accuracy/balanced_accuracy etc. → submit hard labels, NOT probability matrix (S6E4 trap!)

Always cross-check the metric on the competition's Overview page, not the sample file.

S6E4 mirror bug (the inverse of S6E2)

The S6E2 bug was "AUC wanted proba, I sent 0/1". The S6E4 bug is the mirror: accuracy wanted hard labels, I sent a probability matrix.

S6E4 sample_submission.csv:

id,Irrigation_Need
630000,Low
630001,Low

If you submit:

id,High,Low,Medium
630000,0.001,0.999,0.000

Kaggle will reject it (SUBMISSION ERROR) because columns don't match sample.

Fix: use predictor.predict(test) (hard labels) instead of predict_proba() for accuracy-style metrics. Or argmax the probability columns to recover labels.

Both bugs are caught by a 30-line GSD verifier that compares your submission columns to sample_submission.csv columns. See gsd-loop-engineering skill for the full workflow.

Verification

Before submitting:

  • ✅ Read the metric name in the competition's "Evaluation" tab
  • ✅ Check whether the metric is in the "DO NOT threshold" list above
  • ✅ Use predict_proba() (not predict()) when in doubt for classification
  • ✅ For multi-class AUC: column order in submission = column order in sample_submission.csv

Sanity check after first submission:

  • Public LB score is within 0.01-0.02 of OOF (typical for probability submissions)
  • Public LB score is suspiciously low (<0.55 for binary, <1/n_classes for multi-class) → re-check format

Example: S6E2 Heart Disease, what went wrong

# Training (correct)
predictor = TabularPredictor(label='Heart Disease', eval_metric='roc_auc').fit(
    train_data, presets='best_quality', time_limit=900
)
# Note: predictor automatically uses AUC internally, so OOF is reliable

# First submission attempt (WRONG)
preds = predictor.predict(test)            # returns "Presence" / " Absence" labels
# Thresholded to 0/1
submission = pd.DataFrame({'id': test['id'], 'Heart Disease': (preds == 'Presence').astype(int)})
submission.to_csv('submission.csv', index=False)
# → Public LB 0.88403 (looked like total failure)

# Second attempt (CORRECT)
preds_proba = predictor.predict_proba(test)
# preds_proba columns: ['Absence', 'Presence'] (alphabetical)
submission = pd.DataFrame({'id': test['id'], 'Heart Disease': preds_proba['Presence']})
submission.to_csv('submission_proba.csv', index=False)
# → Public LB 0.95357 (matches OOF)

Notes

Why this is a sneaky bug:

  • OOF score looks great, code "works" locally
  • Local validation passes
  • The error only surfaces on the public leaderboard
  • A 0/1 submission's AUC is exactly 0.5 (random) when the model is balanced — easy to mistake for a real overfitting problem and start "fixing" the model

Why this happens more with AutoGluon:

  • predictor.predict() defaults to class labels
  • Other frameworks (sklearn, xgboost) often default to probabilities depending on the API
  • Easy to forget the .predict_proba() distinction

Time investment:

  • Fix: 30 seconds (change predict → predict_proba, save, resubmit)
  • Cost of skipping: 1+ day debugging "why is my model broken"

Related Skills

  • kaggle-data-format-first — input data format (2D vs 3D), complementary concern
  • kaggle-competition-best-practices — overall submission workflow
  • kaggle-optimal-blending — blend probability outputs across models
  • cv-lb-gap-acknowledgment — distinguishes submission-format gaps from real CV-LB gaps
  • autogluon-first — uses predict_proba() correctly for AUC

References

slack-gif-creatoranthropics/skillsKnowledge and utilities for creating animated GIFs optimized for Slack. Provides constraints, validation tools, and animation concepts. Use when users request animated GIFs for Slack like "make me a GIF of X doing Y for Slack."Stars: 198kUpdated Claude CodeCodexVerifiedInstall
ponytailDietrichGebert/ponytailLazy senior dev mode for any coding task: the smallest change that fully solves it, and a reply a busy human understands in one read.Stars: 194kUpdated Claude CodeCodexVerifiedInstall
ponytail-debtDietrichGebert/ponytailHarvest every shortcut: comment into one debt ledger, so deferrals get tracked instead of forgotten. One-shot report.Stars: 194kUpdated Claude CodeCodexVerifiedInstall
ponytail-debtDietrichGebert/ponytailList every `shortcut:` comment (and older `ponytail:` ones) as a debt ledger. One-shot report, changes nothing. Use for "ponytail debt", "what did ponytail defer", "list the shortcuts", /ponytail-debt.Stars: 194kUpdated Claude CodeCodexVerifiedInstall