topprismdata/cultivating-ml-agent / skills/examples/progressive-verification-debugging
progressive-verification-debugging
VerifiedProgressive verification methodology for debugging complex systems. Use when: (1) Facing mysterious crashes or errors with no clear cause, (2) System works in some environments but not others, (3) Multiple potential failure points exist, (4) Need to isolate whether issue is environment configuration vs code bug. Core principle: Start with simplest test, gradually increase complexity to pinpoint exact failure point.
Progressive Verification Methodology
Problem
When debugging complex systems (ML training, distributed systems, integrations), it's easy to:
- Waste time checking logs without a strategy
- Assume the wrong component is broken
- Try complex solutions before understanding root cause
- Get overwhelmed by multiple potential failure points
Context / Trigger Conditions
Use this methodology when:
- Silent crashes with no error messages
- "Works on my machine" but fails elsewhere
- Multiple layers: environment → code → data → model
- Version compatibility issues suspected
- Need to isolate environment vs code issues
Classic symptoms:
- Process exits silently after starting
- GPU utilization 0-1% despite training script running
- Empty log files despite process being active
- Different behavior across platforms (Mac vs WSL2 vs Linux)
Solution
Core Principle
"先要用简单代码验证环境,然后判断是不是配置问题"
Always start with the simplest test, gradually increase complexity.
Progressive Verification Steps
Create a test ladder from simplest to most complex:
Level 1: Basic Infrastructure
├── Test: torch.randn(2, 3).cuda()
└── Validates: CUDA driver, GPU access
Level 2: Core Components
├── Test: nn.Linear(10, 5).cuda()
└── Validates: Model creation, CUDA memory allocation
Level 3: Real Models
├── Test: ResNet50(weights=None).cuda()
└── Validates: Complex model architecture
Level 4: Data Pipeline (Synthetic)
├── Test: DataLoader with random tensors
└── Validates: DataLoader, multiprocessing
Level 5: Data Pipeline (Real)
├── Test: PIL.Image.open + transforms
└── Validates: File I/O, image decoding
Level 6: Training Loop
├── Test: Single forward + backward pass
└── Validates: Optimization, gradient flow
Decision Tree
Does Level N pass?
├── Yes: Proceed to Level N+1
└── No: Issue is at Level N
└── Fix before proceeding
Example Test Script
import torch
import torch.nn as nn
from torch.utils.data import Dataset, DataLoader
from torchvision.models import resnet50
from PIL import Image
print("=== Progressive Verification ===")
# Level 1: Basic CUDA
print("Level 1: Basic CUDA")
x = torch.randn(10, 10).cuda()
print(" PASSED")
# Level 2: Model on CUDA
print("Level 2: Model on CUDA")
model = nn.Linear(10, 5).cuda()
y = model(x)
print(" PASSED")
# Level 3: ResNet50 on CUDA
print("Level 3: ResNet50 on CUDA")
resnet = resnet50(weights=None).cuda()
print(" PASSED")
# Level 4: DataLoader (synthetic)
print("Level 4: DataLoader (synthetic)")
class DummyDataset(Dataset):
def __len__(self): return 10
def __getitem__(self, i): return torch.randn(3, 224, 224), 0
ds = DummyDataset()
loader = DataLoader(ds, batch_size=4, num_workers=0)
for x, y in loader:
x, y = x.cuda(), y.cuda()
output = resnet(x)
break
print(" PASSED")
# Level 5: Real data loading
print("Level 5: Real data loading")
# Test actual file loading here
print(" PASSED")
print("All levels passed!")
Strategy: Isolate Variables
When Level N fails, you know the problem is at that exact level:
| Fail Level | Known Working | Known Broken | Likely Issue |
|---|---|---|---|
| Level 1 | None | CUDA basics | Driver, GPU |
| Level 2 | CUDA | Model creation | Memory, architecture |
| Level 3 | Simple models | Complex models | Model size, dependencies |
| Level 4 | Models | DataLoader | Multiprocessing, I/O |
| Level 5 | Synthetic data | Real data | File format, permissions |
| Level 6 | Data loading | Training loop | Optimizer, gradients |
Common Anti-Patterns
❌ Don't: Run full training script first, then debug when it fails ✅ Do: Verify each component independently before combining
❌ Don't: Check logs without hypothesis ✅ Do: Create minimal test to confirm hypothesis
❌ Don't: Change multiple things at once ✅ Do: Change one variable, test, repeat
Verification
After applying this methodology:
- You should know exactly which level fails
- You should have a minimal reproduction (the failing test)
- You should have eliminated all higher levels as causes
- Fix should be validated by re-running that specific test
Example: WSL2 PyTorch Debug Session
Problem: Training crashes silently after 1-3 batches
Progressive Verification Results:
- Level 1 ✅: Basic CUDA works
- Level 2 ✅: Model on CUDA works
- Level 3 ✅: ResNet50 on CUDA works
- Level 4 ✅: DataLoader with dummy data works
- Level 5 ❌: Real image loading fails with
._train_XXXX.png
Root Cause Identified: macOS metadata files
Fix Applied: Filter out ._ prefix in file listing
Result: Training works after fix
This took 30 minutes with progressive verification vs. hours without it.
Notes
Philosophy
This methodology is based on scientific method:
- Hypothesis: Problem is at level N
- Experiment: Run test for level N
- Observe: Pass or fail
- Conclude: Isolate or eliminate
When to Skip Levels
If you have strong evidence, you can skip levels:
- Previous debugging already ruled out lower levels
- Error message directly points to specific component
- Similar issue was solved before
But always verify your assumption first!
Time Trade-off
Creating tests takes time upfront, but saves time overall:
- Test creation: 10-20 minutes
- Focused debugging: 20-60 minutes
- Total: 30-80 minutes
vs. random debugging:
- Log spelunking: 1-2 hours
- Trial-and-error: 2-4 hours
- Total: 3-6 hours
Adaptability
This pattern applies beyond ML:
- Web services: Database → API → Frontend → Integration
- Data pipelines: Source → Extract → Transform → Load
- DevOps: Local → Staging → Production
- Distributed systems: Single node → Cluster → Network
References
Empirical validation:
- WSL2 PyTorch training crash (2026-03-01): Isolated PIL image loading issue in 30 minutes
- Progressive testing reduced debugging time by 60-80% across multiple sessions
Related concepts:
- Binary search algorithm applied to debugging
- Minimal reproducible example (MRE) principle
- Test-driven development (TDD) mindset
See also:
- wsl2-pytorch-training-crash - Specific application example
- systematic-debugging - General debugging workflow
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| finishing-a-development-branchobra/superpowersUse when implementation is complete, all tests pass, and you need to decide how to integrate the work | Stars: 319k | Updated | Claude CodeCodex | Verified | Install |
| systematic-debuggingobra/superpowersUse when encountering any bug, test failure, or unexpected behavior, before proposing fixes | Stars: 319k | Updated | Claude CodeCodex | Verified | Install |
| test-driven-developmentobra/superpowersUse when implementing any feature or bugfix, before writing implementation code | Stars: 319k | Updated | Claude CodeCodex | Verified | Install |
| skill-creatoranthropics/skillsCreate new skills, modify and improve existing skills, and measure skill performance. Use when users want to create a skill from scratch, edit, or optimize an existing skill, run evals to test a skill, benchmark skill performance with variance analysis, or optimize a skill's description for better triggering accuracy. | Stars: 197k | Updated | Claude CodeCodex | Verified | Install |