Orchestra-Research/AI-Research-SKILLs / 10-optimization/ml-training-recipes
ml-training-recipes
VerifiedBattle-tested PyTorch training recipes for all domains — LLMs, vision, diffusion, medical imaging, protein/drug discovery, spatial omics, genomics. Covers training loops, optimizer selection (AdamW, Muon), LR scheduling, mixed precision, debugging, and systematic experimentation. Use when training or fine-tuning neural networks, debugging loss spikes or OOM, choosing architectures, or optimizing GPU throughput.
| File | Kind | Size |
|---|---|---|
| SKILL.md | skill md | 11.2 KB |
| references/architecture.md | markdown | 10.3 KB |
| references/biomedical.md | markdown | 20.8 KB |
| references/domain-specific.md | markdown | 19.2 KB |
| references/experiment-loop.md | markdown | 4.5 KB |
| references/optimizers.md | markdown | 10.6 KB |
| references/scaling-and-selection.md | markdown | 16.9 KB |
references/experiment-loop.md
View on GitHub# Autonomous Experiment Loop (autoresearch pattern)
A systematic workflow for rapid ML experimentation, drawn from Karpathy's autoresearch project.
Use this when iterating on architecture or hyperparameters and you want to run many quick experiments.
## Core idea
Run every experiment with a **fixed time budget** (e.g., 5 minutes) so results are directly comparable.
This enables ~12 experiments/hour or ~100 overnight. The key insight: wall-clock time is a better
budget unit than steps or epochs because it naturally accounts for throughput differences between configs.
## The experiment loop
```
1. Read current state (results.tsv, train.py)
2. Decide what to try next (one change at a time)
3. Modify train.py
4. git commit -m "description of change"
5. Run training (with timeout)
6. Parse results from stdout
7. Decision:
- If val_bpb improved → KEEP (advance branch)
- If val_bpb worsened → DISCARD (git reset --hard HEAD~1)
- If crashed → FIX trivial bugs and retry, or LOG and move on
8. Append result to results.tsv
9. Repeat
```
## Results tracking
```
commit val_bpb memory_gb status description
a1b2c3d 0.9979 44.0 keep baseline
b2c3d4e 0.9932 44.2 keep increase matrix LR to 0.04
c3d4e5f 1.0050 44.0 discard switch to GeLU activation
d4e5f6g 0.0000 0.0 crash double model width (OOM)
```
## Key principles
### Single-file constraint
Confine all changes to one file (e.g., `train.py`). This makes diffs reviewable and rollbacks clean.
Everything — model, optimizer, data loading, evaluation — lives in one file during experimentation.
Refactor into modules only after the experiment phase.
### Keep/discard discipline
- **Keep**: val metric improved (or equal with less memory/time)
- **Discard**: val metric worsened, regardless of how clever the idea was
- **The simplicity criterion**: all else being equal, simpler is better. Removing something and
getting equal results is a great outcome — it means the removed thing was dead weight.
### Crash recovery
- **Trivial crash** (typo, shape mismatch): fix and retry the same experiment
- **Fundamental crash** (OOM, numerical instability): log as `crash`, move on
- **Timeout** (>2x budget): kill the process, log as `timeout`
### Fixed budget comparison
```python
import time
TIME_BUDGET = 300 # 5 minutes
t_start = time.time()
for step in range(max_steps):
# ... training step ...
elapsed = time.time() - t_start
if elapsed >= TIME_BUDGET:
break
```
## Tokenizer training
When training from scratch, train a BPE tokenizer on your data:
```python
import rustbpe
# GPT-4 split pattern (handles code, numbers, whitespace well)
SPLIT_PATTERN = r"""'(?i:[sdmt]|ll|ve|re)|[^\r\n\p{L}\p{N}]?+\p{L}+|\p{N}{1,3}| ?[^\s\p{L}\p{N}]++[\r\n]*|\s*[\r\n]|\s+(?!\S)|\s+"""
# Train tokenizer
tokenizer = rustbpe.Tokenizer()
tokenizer.train(
text_iterator, # yields text chunks
vocab_size=8192, # small vocab for quick experiments; 32K+ for production
split_pattern=SPLIT_PATTERN,
special_tokens=["<|bos|>"]
)
# Build token_bytes lookup for BPB evaluation
token_bytes = torch.zeros(vocab_size, dtype=torch.long)
for i in range(vocab_size):
token_bytes[i] = len(tokenizer.decode([i]).encode("utf-8"))
```
### Vocab size tradeoffs
| Vocab Size | Use Case | Notes |
|-----------|----------|-------|
| 4K-8K | Quick experiments, small models | Faster tokenizer training, more tokens per doc |
| 32K | Standard LLM pretraining | Good balance of compression and vocab coverage |
| 64K-128K | Multilingual, code-heavy | Better compression but larger embedding table |
## Data preparation
### Shard-based train/val split
```python
# Use last shard as validation (always the same data for consistent eval)
shard_files = sorted(glob("data/shard_*.bin"))
val_shard = shard_files[-1] # pinned validation
train_shards = shard_files[:-1] # everything else
```
Split by shard, not by random sampling — this ensures no data leakage and makes
the val set deterministic across experiments.
## Environment setup
```python
import os
os.environ["PYTORCH_ALLOC_CONF"] = "expandable_segments:True" # BEFORE torch import
os.environ["HF_HUB_DISABLE_PROGRESS_BARS"] = "1" # clean logs
import torch
```
Setting `PYTORCH_ALLOC_CONF` before importing torch is important — it configures the
CUDA allocator at initialization time.
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| slack-gif-creatoranthropics/skillsKnowledge and utilities for creating animated GIFs optimized for Slack. Provides constraints, validation tools, and animation concepts. Use when users request animated GIFs for Slack like "make me a GIF of X doing Y for Slack." | Stars: 198k | Updated | Claude CodeCodex | Verified | Install |
| ponytailDietrichGebert/ponytailLazy senior dev mode for any coding task: the smallest change that fully solves it, and a reply a busy human understands in one read. | Stars: 194k | Updated | Claude CodeCodex | Verified | Install |
| ponytail-debtDietrichGebert/ponytailHarvest every shortcut: comment into one debt ledger, so deferrals get tracked instead of forgotten. One-shot report. | Stars: 194k | Updated | Claude CodeCodex | Verified | Install |
| ponytail-debtDietrichGebert/ponytailList every `shortcut:` comment (and older `ponytail:` ones) as a debt ledger. One-shot report, changes nothing. Use for "ponytail debt", "what did ponytail defer", "list the shortcuts", /ponytail-debt. | Stars: 194k | Updated | Claude CodeCodex | Verified | Install |