Skip to content

topprismdata/cultivating-ml-agent / skills/examples/controlled-submission-experiment

controlled-submission-experiment

Verified

Use when: (1) a new submission scores worse than baseline and the reason is unclear, (2) multiple changes were made simultaneously (new model + new post-processing + new features), (3) need to isolate which component caused a regression, (4) CV improves but LB degrades, (5) comparing "smart" vs "simple" approaches, (6) verifying that model improvements are real vs artifacts of post-processing changes. CRITICAL: Never change more than one variable between submissions without controlled comparison.

Controlled Submission Experiment

Problem

When multiple changes are bundled into a single submission (new features + new model + new post-processing), it's impossible to tell which change helped or hurt. A submission that scores worse than baseline might actually contain valuable model improvements hidden by a post-processing regression.

Symptoms

  • New submission scores worse than baseline despite "better" CV
  • Multiple changes made simultaneously (features + model + post-processing)
  • Zero ratio in submission changes significantly (>2%) between versions
  • Can't explain WHY the score changed in a specific direction

Solution

The Controlled Experiment Pattern

Isolate one variable at a time by creating multiple submissions from the same model output:

Baseline:  Model_A + Postproc_A  → Score_A
New:       Model_B + Postproc_B  → Score_B
Control:   Model_B + Postproc_A  → Score_C  (KEY!)

Interpretation matrix:

PatternModel BPostproc BConclusion
Score_C > Score_A, Score_B ≈ Score_CBetterNeutralModel B is better, postproc doesn't matter
Score_C > Score_A, Score_C >> Score_BBetterWorseModel B is better BUT postproc B regresses
Score_C < Score_A, Score_B ≈ Score_CWorseNeutralModel B is genuinely worse
Score_C ≈ Score_A, Score_B >> Score_ANeutralBetterPostproc B carries the improvement
Score_C > Score_A, Score_B > Score_CBetterBetterBoth improve, model B slightly more

Implementation

import pandas as pd
import numpy as np

# Step 1: Load both submission predictions (before post-processing)
r10_raw = pd.read_csv("submission_r10_raw.csv")  # baseline raw predictions
r11b_raw = pd.read_csv("submission_r11b_raw.csv")  # new raw predictions

# Step 2: Identify disputed predictions
merged = r10_raw.merge(r11b_raw, on="id", suffixes=("_r10", "_r11b"))
disputed = merged[
    (merged["sales_r11b"] == 0) & (merged["sales_r10"] > 0)
]
print(f"Disputed rows: {len(disputed)}")
print(f"R10 values in disputed: mean={disputed['sales_r10'].mean():.2f}")

# Step 3: Create controlled submission
# Use Model_B predictions but with Postproc_A logic
controlled = r11b_raw.copy()
# Apply only the baseline post-processing
controlled.loc[controlled["sales"] < 0.1, "sales"] = 0
controlled.to_csv("submission_r11c_controlled.csv", index=False)

Diagnostic: Zero Ratio Comparison

# Quick diagnostic — always check this
for name, df in [("R10", r10), ("R11b", r11b), ("R11c", r11c)]:
    zero_pct = (df["sales"] == 0).mean()
    print(f"{name}: zeros={zero_pct:.2%}, mean={df['sales'].mean():.2f}")

# If zero ratio jumps >2% between versions, investigate immediately

Prevention

  1. One variable at a time — never change model AND post-processing in the same submission
  2. Always compare zero ratios — a jump >2% signals a post-processing change that may hurt
  3. Save raw predictions before post-processing, so you can re-apply different post-processing
  4. Name your experiments (R10, R11b, R11c) to track what changed between each

Evidence (Real Experiment)

Kaggle Store Sales (Favorita), April 2026:

VersionModelPost-processingZerosLB
R10baseline< 0.1 → 08.36%0.39880
R11bimprovedmin_sales adaptive12.41%0.40073
R11cimproved< 0.1 → 08.30%0.39824

Without the controlled experiment, we would have concluded "the improved model is worse." With it, we proved the model was actually +0.00056 better, but post-processing regressed -0.00249.

Notes

  • This is the ML equivalent of A/B testing — isolate variables rigorously
  • Related to rmsle-zero-threshold-asymmetry — the specific post-processing failure mode
  • Related to kaggle-optimal-blending — when blending, also test components in isolation
  • For ensemble experiments, use the same pattern: add one model at a time, measure delta
brainstormingobra/superpowersYou MUST use this before any creative work - creating features, building components, adding functionality, modifying behavior, or planning anything new, in software or out of it (a talk, a business, a renovation).Stars: 322kUpdated Claude CodeCodexVerifiedInstall
brand-guidelinesanthropics/skillsApplies Anthropic's official brand colors and typography to any sort of artifact that may benefit from having Anthropic's look-and-feel. Use it when brand colors or style guidelines, visual formatting, or company design standards apply.Stars: 198kUpdated Claude CodeCodexVerifiedInstall
frontend-designanthropics/skillsGuidance for distinctive, intentional visual design when building new UI or reshaping an existing one. Helps with aesthetic direction, typography, and making choices that don't read as templated defaults.Stars: 198kUpdated Claude CodeCodexVerifiedInstall
mcp-builderanthropics/skillsGuide for creating high-quality MCP (Model Context Protocol) servers that enable LLMs to interact with external services through well-designed tools. Use when building MCP servers to integrate external APIs or services, whether in Python (FastMCP) or Node/TypeScript (MCP SDK).Stars: 198kUpdated Claude CodeCodexVerifiedInstall