Skip to content

topprismdata/cultivating-ml-agent / skills/examples/progressive-verification-debugging

progressive-verification-debugging

Verified

Progressive verification methodology for debugging complex systems. Use when: (1) Facing mysterious crashes or errors with no clear cause, (2) System works in some environments but not others, (3) Multiple potential failure points exist, (4) Need to isolate whether issue is environment configuration vs code bug. Core principle: Start with simplest test, gradually increase complexity to pinpoint exact failure point.

Progressive Verification Methodology

Problem

When debugging complex systems (ML training, distributed systems, integrations), it's easy to:

  • Waste time checking logs without a strategy
  • Assume the wrong component is broken
  • Try complex solutions before understanding root cause
  • Get overwhelmed by multiple potential failure points

Context / Trigger Conditions

Use this methodology when:

  • Silent crashes with no error messages
  • "Works on my machine" but fails elsewhere
  • Multiple layers: environment → code → data → model
  • Version compatibility issues suspected
  • Need to isolate environment vs code issues

Classic symptoms:

  • Process exits silently after starting
  • GPU utilization 0-1% despite training script running
  • Empty log files despite process being active
  • Different behavior across platforms (Mac vs WSL2 vs Linux)

Solution

Core Principle

"先要用简单代码验证环境,然后判断是不是配置问题"

Always start with the simplest test, gradually increase complexity.

Progressive Verification Steps

Create a test ladder from simplest to most complex:

Level 1: Basic Infrastructure
├── Test: torch.randn(2, 3).cuda()
└── Validates: CUDA driver, GPU access

Level 2: Core Components
├── Test: nn.Linear(10, 5).cuda()
└── Validates: Model creation, CUDA memory allocation

Level 3: Real Models
├── Test: ResNet50(weights=None).cuda()
└── Validates: Complex model architecture

Level 4: Data Pipeline (Synthetic)
├── Test: DataLoader with random tensors
└── Validates: DataLoader, multiprocessing

Level 5: Data Pipeline (Real)
├── Test: PIL.Image.open + transforms
└── Validates: File I/O, image decoding

Level 6: Training Loop
├── Test: Single forward + backward pass
└── Validates: Optimization, gradient flow

Decision Tree

Does Level N pass?
├── Yes: Proceed to Level N+1
└── No: Issue is at Level N
    └── Fix before proceeding

Example Test Script

import torch
import torch.nn as nn
from torch.utils.data import Dataset, DataLoader
from torchvision.models import resnet50
from PIL import Image

print("=== Progressive Verification ===")

# Level 1: Basic CUDA
print("Level 1: Basic CUDA")
x = torch.randn(10, 10).cuda()
print("  PASSED")

# Level 2: Model on CUDA
print("Level 2: Model on CUDA")
model = nn.Linear(10, 5).cuda()
y = model(x)
print("  PASSED")

# Level 3: ResNet50 on CUDA
print("Level 3: ResNet50 on CUDA")
resnet = resnet50(weights=None).cuda()
print("  PASSED")

# Level 4: DataLoader (synthetic)
print("Level 4: DataLoader (synthetic)")
class DummyDataset(Dataset):
    def __len__(self): return 10
    def __getitem__(self, i): return torch.randn(3, 224, 224), 0

ds = DummyDataset()
loader = DataLoader(ds, batch_size=4, num_workers=0)
for x, y in loader:
    x, y = x.cuda(), y.cuda()
    output = resnet(x)
    break
print("  PASSED")

# Level 5: Real data loading
print("Level 5: Real data loading")
# Test actual file loading here
print("  PASSED")

print("All levels passed!")

Strategy: Isolate Variables

When Level N fails, you know the problem is at that exact level:

Fail LevelKnown WorkingKnown BrokenLikely Issue
Level 1NoneCUDA basicsDriver, GPU
Level 2CUDAModel creationMemory, architecture
Level 3Simple modelsComplex modelsModel size, dependencies
Level 4ModelsDataLoaderMultiprocessing, I/O
Level 5Synthetic dataReal dataFile format, permissions
Level 6Data loadingTraining loopOptimizer, gradients

Common Anti-Patterns

❌ Don't: Run full training script first, then debug when it fails ✅ Do: Verify each component independently before combining

❌ Don't: Check logs without hypothesis ✅ Do: Create minimal test to confirm hypothesis

❌ Don't: Change multiple things at once ✅ Do: Change one variable, test, repeat

Verification

After applying this methodology:

  1. You should know exactly which level fails
  2. You should have a minimal reproduction (the failing test)
  3. You should have eliminated all higher levels as causes
  4. Fix should be validated by re-running that specific test

Example: WSL2 PyTorch Debug Session

Problem: Training crashes silently after 1-3 batches

Progressive Verification Results:

  • Level 1 ✅: Basic CUDA works
  • Level 2 ✅: Model on CUDA works
  • Level 3 ✅: ResNet50 on CUDA works
  • Level 4 ✅: DataLoader with dummy data works
  • Level 5 ❌: Real image loading fails with ._train_XXXX.png

Root Cause Identified: macOS metadata files

Fix Applied: Filter out ._ prefix in file listing

Result: Training works after fix

This took 30 minutes with progressive verification vs. hours without it.

Notes

Philosophy

This methodology is based on scientific method:

  1. Hypothesis: Problem is at level N
  2. Experiment: Run test for level N
  3. Observe: Pass or fail
  4. Conclude: Isolate or eliminate

When to Skip Levels

If you have strong evidence, you can skip levels:

  • Previous debugging already ruled out lower levels
  • Error message directly points to specific component
  • Similar issue was solved before

But always verify your assumption first!

Time Trade-off

Creating tests takes time upfront, but saves time overall:

  • Test creation: 10-20 minutes
  • Focused debugging: 20-60 minutes
  • Total: 30-80 minutes

vs. random debugging:

  • Log spelunking: 1-2 hours
  • Trial-and-error: 2-4 hours
  • Total: 3-6 hours

Adaptability

This pattern applies beyond ML:

  • Web services: Database → API → Frontend → Integration
  • Data pipelines: Source → Extract → Transform → Load
  • DevOps: Local → Staging → Production
  • Distributed systems: Single node → Cluster → Network

References

Empirical validation:

  • WSL2 PyTorch training crash (2026-03-01): Isolated PIL image loading issue in 30 minutes
  • Progressive testing reduced debugging time by 60-80% across multiple sessions

Related concepts:

  • Binary search algorithm applied to debugging
  • Minimal reproducible example (MRE) principle
  • Test-driven development (TDD) mindset

See also:

finishing-a-development-branchobra/superpowersUse when implementation is complete, all tests pass, and you need to decide how to integrate the workStars: 319kUpdated Claude CodeCodexVerifiedInstall
systematic-debuggingobra/superpowersUse when encountering any bug, test failure, or unexpected behavior, before proposing fixesStars: 319kUpdated Claude CodeCodexVerifiedInstall
test-driven-developmentobra/superpowersUse when implementing any feature or bugfix, before writing implementation codeStars: 319kUpdated Claude CodeCodexVerifiedInstall
skill-creatoranthropics/skillsCreate new skills, modify and improve existing skills, and measure skill performance. Use when users want to create a skill from scratch, edit, or optimize an existing skill, run evals to test a skill, benchmark skill performance with variance analysis, or optimize a skill's description for better triggering accuracy.Stars: 197kUpdated Claude CodeCodexVerifiedInstall