Skip to content

topprismdata/cultivating-ml-agent / skills/examples/kaggle-data-format-first

kaggle-data-format-first

Unverified

Prevent wasted research by verifying Kaggle competition data format BEFORE investing in RAG, technical planning, or model architecture design. Use when: (1) Starting any new Kaggle competition, (2) Competition name/size is ambiguous about data format, (3) Planning to do extensive research before implementation, (4) Download size doesn't match expected data structure. Critical for competitions where name suggests one format (e.g., "3D Surface Detection") but actual data is different (e.g., 2D images).

Kaggle Data Format Verification Before Research

Problem

Competition names and file sizes can be misleading. Investing in RAG research, technical planning, or model architecture design before verifying the actual data format leads to significant wasted effort when assumptions don't match reality.

Real example:

  • Competition: "vesuvius-challenge-surface-detection"
  • Expected (from RAG): 3D TIFF stacks, TopoScore, 128³ patches
  • Actual data: 2D grayscale images (320×320), binary masks
  • Waste: Hours of RAG research on wrong problem

Context / Trigger Conditions

Use this skill when:

  • Starting ANY new Kaggle competition
  • Competition name is ambiguous about data dimensionality (2D vs 3D)
  • Data size suggests one format but could be another
  • Planning to do RAG research or extensive technical planning
  • File extensions are generic (.tif, .png, .npy could be anything)

Red flags:

  • Competition name mentions "3D", "volume", "surface" but you haven't verified
  • Large download size (>5GB) but unsure what format it actually is
  • Multiple data directories with unclear purpose (train_images vs train vs train_data)

Solution

Phase 1: Quick Format Check (Before ANY Research)

Step 1: Download only a sample first

# If possible, download just one file to verify format
# Or download full data but check structure immediately

kaggle competitions download -c {competition-slug}
unzip {competition-file}.zip

Step 2: Verify data structure in <5 minutes

import os
from PIL import Image
import numpy as np

# Quick check script
data_dir = "path/to/unzipped/data"

# What files exist?
print("Directories:", [d for d in os.listdir(data_dir) if os.path.isdir(os.path.join(data_dir, d))])
print("Files:", [f for f in os.listdir(data_dir) if os.path.isfile(os.path.join(data_dir, f))][:10])

# Check dimensions of first sample
samples = []
for root, dirs, files in os.walk(data_dir):
    for f in files:
        if f.endswith(('.tif', '.png', '.jpg', '.npy')):
            path = os.path.join(root, f)
            if f.endswith('.npy'):
                data = np.load(path)
            else:
                data = np.array(Image.open(path))

            print(f"Sample: {f}")
            print(f"  Shape: {data.shape}")
            print(f"  Dtype: {data.dtype}")
            print(f"  Range: [{data.min()}, {data.max()}]")

            samples.append({
                'path': path,
                'shape': data.shape,
                'dtype': str(data.dtype)
            })

            if len(samples) >= 3:
                break
    if len(samples) >= 3:
        break

# Determine data type
if all(len(s['shape']) == 2 for s in samples):
    print("✓ Data Type: 2D Images")
elif all(len(s['shape']) == 3 for s in samples):
    print("✓ Data Type: 3D Volumes")
else:
    print("⚠ Mixed or irregular dimensions")

Step 3: Confirm with competition description

# Read competition description on Kaggle
# Check Data tab for file descriptions
# Look at sample notebooks from other participants

Phase 2: Match Format to Approach

Data FormatTypical ApproachRed Flags to Avoid
2D ImagesU-Net 2D, ResNet, segmentation❌ Using 3D architectures
3D VolumesnnU-Net 3D, 3D U-Net, V-Net❌ Flattening to 2D loses context
TabularXGBoost, LightGBM, CatBoost❌ Using CNN/RNN unnecessarily
Time SeriesLSTM, GRU, Temporal Fusion❌ Treating as i.i.d. samples
TextTransformers, BERT, RoBERTa❌ Using bag-of-words

Phase 3: Proceed with Confidence

Only AFTER verifying data format:

  1. Do RAG research with correct context
  2. Design appropriate architecture
  3. Plan training strategy
  4. Set up evaluation metrics

Verification

Success criteria:

  • ✅ You know exact dimensions (2D vs 3D vs 4D)
  • ✅ You know data type (image, volume, tabular, text)
  • ✅ You know number of channels/samples
  • ✅ You verified labels match expectations
  • ✅ Research/planning now matches actual data

Failure signs (means you need to re-verify):

  • ❌ Using 3D CNNs on 2D data
  • ❌ Researching TopoScore for binary masks
  • ❌ Planning patch extraction for already-small images
  • ❌ RAG answers don't match your data structure

Example

Bad workflow (what happened):

  1. ❌ Saw competition name "vesuvius-challenge-surface-detection"
  2. ❌ Assumed 3D based on name
  3. ❌ Did extensive RAG research on 3D surface detection
  4. ❌ Created technical plan for 3D nnU-Net
  5. ❌ Downloaded 24GB data
  6. ❌ Discovered data is 2D segmentation
  7. ❌ Wasted hours on wrong approach

Good workflow (what should have happened):

  1. ✅ Saw competition name
  2. ✅ Downloaded data FIRST (5 minutes)
  3. ✅ Ran quick format check script (2 minutes)
  4. ✅ Discovered: 2D grayscale (320×320), binary masks
  5. ✅ Adjusted research query: "Vesuvius Challenge 2D segmentation"
  6. ✅ Created appropriate 2D U-Net plan
  7. ✅ Total time saved: Several hours

Notes

Why this happens:

  • Competition names are marketing, not technical specs
  • "Surface Detection" could mean 2D edge detection OR 3D surface reconstruction
  • File size (24GB) could be: many 2D images OR fewer 3D volumes OR compressed videos
  • Multiple Kaggle competitions may have similar names but different tasks

Time investment:

  • Format check: 5-10 minutes
  • Cost of skipping it: 2-10 hours of wasted research

Integration with RAG:

  • Always verify data format FIRST
  • Then use verified format in RAG queries
  • Example: "2D grayscale segmentation for papyrus scrolls" NOT "Vesuvius Challenge 3D"

Related skills:

  • kaggle-competition-best-practices - Overall workflow
  • kaggle-top-performer-replication - After format is verified
  • kaggle-reid-submission-workflow - Task-specific workflows

References

brainstormingobra/superpowersYou MUST use this before any creative work - creating features, building components, adding functionality, modifying behavior, or planning anything new, in software or out of it (a talk, a business, a renovation).Stars: 319kUpdated Claude CodeCodexVerifiedInstall
brand-guidelinesanthropics/skillsApplies Anthropic's official brand colors and typography to any sort of artifact that may benefit from having Anthropic's look-and-feel. Use it when brand colors or style guidelines, visual formatting, or company design standards apply.Stars: 196kUpdated Claude CodeCodexVerifiedInstall
frontend-designanthropics/skillsGuidance for distinctive, intentional visual design when building new UI or reshaping an existing one. Helps with aesthetic direction, typography, and making choices that don't read as templated defaults.Stars: 196kUpdated Claude CodeCodexVerifiedInstall
mcp-builderanthropics/skillsGuide for creating high-quality MCP (Model Context Protocol) servers that enable LLMs to interact with external services through well-designed tools. Use when building MCP servers to integrate external APIs or services, whether in Python (FastMCP) or Node/TypeScript (MCP SDK).Stars: 196kUpdated Claude CodeCodexVerifiedInstall