Orchestra-Research/AI-Research-SKILLs / 06-post-training/torchforge
torchforge-rl-training
VerifiedProvides guidance for PyTorch-native agentic RL using torchforge, Meta's library separating infra from algorithms. Use when you want clean RL abstractions, easy algorithm experimentation, or scalable training with Monarch and TorchTitan.
| File | Kind | Size |
|---|---|---|
| SKILL.md | skill md | 10.5 KB |
| references/api-reference.md | markdown | 8.5 KB |
| references/troubleshooting.md | markdown | 6.6 KB |
references/troubleshooting.md
View on GitHub# torchforge Troubleshooting Guide
## GPU Resource Issues
### Issue: Not Enough GPUs
**Symptoms**: "Insufficient GPU resources" error
**Solutions**:
1. **Reduce service requirements**:
```yaml
services:
generator:
procs: 1
with_gpus: true
trainer:
procs: 1
with_gpus: true
# Remove ref_model or use CPU
```
2. **Use CPU for reference model**:
```yaml
ref_model:
with_gpus: false # Run on CPU
```
3. **Share resources between services**:
```yaml
services:
generator:
procs: 1
num_replicas: 1
colocate_with: trainer # Share GPU with trainer
```
### Issue: Minimum GPU Requirements
**Reference**:
- SFT: 2+ GPUs (trainer + generator)
- GRPO: 3+ GPUs (trainer + generator + ref_model)
- Large models: 8+ GPUs with tensor parallelism
## Memory Issues
### Issue: OOM During Generation
**Symptoms**: CUDA OOM in vLLM
**Solutions**:
1. **Reduce batch size**:
```yaml
grpo:
n_samples: 4 # Reduce from 8
```
2. **Reduce sequence length**:
```yaml
training:
seq_len: 2048 # Reduce from 4096
```
3. **Reduce vLLM memory**:
```yaml
generator:
gpu_memory_utilization: 0.7 # Reduce from 0.9
```
### Issue: OOM During Training
**Symptoms**: CUDA OOM in backward pass
**Solutions**:
1. **Enable gradient checkpointing**:
```yaml
training:
gradient_checkpointing: true
```
2. **Increase gradient accumulation**:
```yaml
training:
gradient_accumulation_steps: 8 # Increase from 4
```
3. **Reduce batch size**:
```yaml
training:
batch_size: 2 # Reduce from 4
```
## Weight Synchronization Issues
### Issue: Slow Weight Sync
**Symptoms**: Long pauses between training and generation
**Solutions**:
1. **Enable RDMA** (if available):
```bash
export TORCHSTORE_USE_RDMA=1
```
2. **Reduce sync frequency**:
```yaml
training:
sync_interval: 10 # Sync every 10 steps
```
3. **Use colocated services**:
```yaml
services:
generator:
colocate_with: trainer
```
### Issue: Weight Sync Failures
**Symptoms**: Errors in weight transfer, stale weights
**Solutions**:
1. **Check network connectivity**:
```bash
ping other_node
```
2. **Increase timeout**:
```yaml
services:
weight_sync_timeout: 600 # 10 minutes
```
3. **Enable sync verification**:
```yaml
training:
verify_weight_sync: true
```
## Training Stability Issues
### Issue: Policy Collapse
**Symptoms**: Entropy drops to zero, reward stops improving
**Solutions**:
1. **Increase KL penalty**:
```yaml
grpo:
beta: 0.2 # Increase from 0.1
```
2. **Add entropy bonus**:
```yaml
training:
entropy_coef: 0.01
```
3. **Reduce learning rate**:
```yaml
training:
learning_rate: 5e-7 # Reduce from 1e-6
```
### Issue: Loss Spikes
**Symptoms**: Sudden loss increases, training instability
**Solutions**:
1. **Enable gradient clipping**:
```yaml
training:
max_grad_norm: 1.0
```
2. **Reduce clip range**:
```yaml
grpo:
clip_low: 0.1 # Reduce from 0.2
clip_high: 0.18 # Reduce from 0.28
```
3. **Use learning rate warmup**:
```yaml
training:
warmup_steps: 100
```
### Issue: Divergent Training
**Symptoms**: Loss becomes NaN, model outputs garbage
**Solutions**:
1. **Check for data issues**:
```python
# Verify no empty sequences
for batch in dataset:
assert batch.input_ids.numel() > 0
```
2. **Use BF16 instead of FP16**:
```yaml
training:
dtype: bfloat16
```
3. **Reduce learning rate significantly**:
```yaml
training:
learning_rate: 1e-7
```
## Service Issues
### Issue: Service Startup Failures
**Symptoms**: Services fail to initialize
**Solutions**:
1. **Check resource availability**:
```bash
nvidia-smi # Verify GPU availability
```
2. **Increase startup timeout**:
```yaml
services:
startup_timeout: 600
```
3. **Check model path**:
```python
from transformers import AutoModelForCausalLM
model = AutoModelForCausalLM.from_pretrained("model_path") # Verify accessible
```
### Issue: Generator Not Responding
**Symptoms**: Generation hangs, timeouts
**Solutions**:
1. **Check vLLM status**:
```python
# Add health check
await generator.health_check.route()
```
2. **Restart service**:
```python
await generator.restart.fanout()
```
3. **Reduce concurrent requests**:
```yaml
generator:
max_concurrent_requests: 10
```
## Monarch Issues
### Issue: Monarch Actor Failures
**Symptoms**: Actor crashes, communication errors
**Solutions**:
1. **Enable fault tolerance**:
```yaml
monarch:
fault_tolerance: true
max_restarts: 3
```
2. **Increase actor memory**:
```yaml
services:
actor_memory_mb: 4096
```
3. **Check Monarch logs**:
```bash
export MONARCH_LOG_LEVEL=DEBUG
```
### Issue: Deadlock in Distributed Communication
**Symptoms**: Training hangs, no progress
**Solutions**:
1. **Check for blocking calls**:
```python
# Use async/await correctly
result = await service.method.route(args) # Correct
# result = service.method.route(args).wait() # May deadlock
```
2. **Add timeouts**:
```python
result = await asyncio.wait_for(
service.method.route(args),
timeout=60.0
)
```
## Installation Issues
### Issue: PyTorch Version Mismatch
**Symptoms**: Import errors, CUDA errors
**Solutions**:
1. **Use provided install script**:
```bash
./scripts/install.sh
```
2. **Verify versions**:
```python
import torch
print(torch.__version__) # Should be 2.9.0+
```
3. **Clean reinstall**:
```bash
pip uninstall torch torchvision torchaudio
./scripts/install.sh
```
### Issue: Monarch Installation Fails
**Symptoms**: Cannot import monarch
**Solutions**:
1. **Install from source**:
```bash
git clone https://github.com/meta-pytorch/monarch
cd monarch && pip install -e .
```
2. **Check CUDA compatibility**:
```bash
nvcc --version # Should match PyTorch CUDA
```
## Debugging Tips
### Enable Verbose Logging
```bash
export FORGE_DEBUG=1
export MONARCH_LOG_LEVEL=DEBUG
```
### Profile Services
```python
# Add profiling
with torch.profiler.profile() as prof:
result = await trainer.train_step.route(batch)
prof.export_chrome_trace("trace.json")
```
### Monitor GPU Utilization
```bash
watch -n 1 nvidia-smi
```
### Test Services Individually
```python
# Test generator
completions = await generator.generate.route(
prompts=["Hello"],
max_tokens=10,
)
print(completions[0].text)
# Test trainer
result = await trainer.train_step.route(dummy_batch)
print(result.loss)
```
## Experimental Warning
Both Monarch and torchforge are experimental. Expect:
- API changes between versions
- Incomplete features
- Bugs in edge cases
Check Discord for latest updates and workarounds.
## Resources
- GitHub Issues: https://github.com/meta-pytorch/torchforge/issues
- Discord: https://discord.gg/YsTYBh6PD9
- Monarch Issues: https://github.com/meta-pytorch/monarch/issues
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| reverse-engineer-anythingmorluto/reaReverse engineer native, managed, Electron/JavaScript, packaged, firmware, and browser targets with REA, and build agent environments (AgentEnv bundles) of web applications from a live instance or a capture, and of Linux desktop apps with a task from a request. Use shipped-artifact or requested runtime evidence to explain features, compare versions, decompile code, or guide a reconstruction. Skip REA for ordinary source-repository architecture analysis. | Stars: 472k | Updated | Claude CodeCodex | Verified | Install |
| diagnosing-superpowersobra/superpowersUse when a superpowers session went wrong and your human partner wants to know why — repeated work, ignored plans, stumbles, poor results, a skill that didn't fire, "it took too long", "why is it so expensive", "what is it doing" — or wants to build a bug report for the superpowers maintainers, for the current session or a past one identified by id or path, on any harness. | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |
| dispatching-parallel-agentsobra/superpowersUse when facing 2+ independent tasks that can be worked on without shared state or sequential dependencies | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |
| executing-plansobra/superpowersUse when executing an implementation plan in the current session as the implementer yourself — your human partner chose inline execution, or no subagent tool is available | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |