Skip to content

Orchestra-Research/AI-Research-SKILLs / 06-post-training/torchforge

torchforge-rl-training

Verified

Provides guidance for PyTorch-native agentic RL using torchforge, Meta's library separating infra from algorithms. Use when you want clean RL abstractions, easy algorithm experimentation, or scalable training with Monarch and TorchTitan.

FileKindSize
SKILL.mdskill md10.5 KB
references/api-reference.mdmarkdown8.5 KB
references/troubleshooting.mdmarkdown6.6 KB

references/troubleshooting.md

View on GitHub
# torchforge Troubleshooting Guide

## GPU Resource Issues

### Issue: Not Enough GPUs

**Symptoms**: "Insufficient GPU resources" error

**Solutions**:

1. **Reduce service requirements**:
```yaml
services:
  generator:
    procs: 1
    with_gpus: true
  trainer:
    procs: 1
    with_gpus: true
  # Remove ref_model or use CPU
```

2. **Use CPU for reference model**:
```yaml
ref_model:
  with_gpus: false  # Run on CPU
```

3. **Share resources between services**:
```yaml
services:
  generator:
    procs: 1
    num_replicas: 1
    colocate_with: trainer  # Share GPU with trainer
```

### Issue: Minimum GPU Requirements

**Reference**:
- SFT: 2+ GPUs (trainer + generator)
- GRPO: 3+ GPUs (trainer + generator + ref_model)
- Large models: 8+ GPUs with tensor parallelism

## Memory Issues

### Issue: OOM During Generation

**Symptoms**: CUDA OOM in vLLM

**Solutions**:

1. **Reduce batch size**:
```yaml
grpo:
  n_samples: 4  # Reduce from 8
```

2. **Reduce sequence length**:
```yaml
training:
  seq_len: 2048  # Reduce from 4096
```

3. **Reduce vLLM memory**:
```yaml
generator:
  gpu_memory_utilization: 0.7  # Reduce from 0.9
```

### Issue: OOM During Training

**Symptoms**: CUDA OOM in backward pass

**Solutions**:

1. **Enable gradient checkpointing**:
```yaml
training:
  gradient_checkpointing: true
```

2. **Increase gradient accumulation**:
```yaml
training:
  gradient_accumulation_steps: 8  # Increase from 4
```

3. **Reduce batch size**:
```yaml
training:
  batch_size: 2  # Reduce from 4
```

## Weight Synchronization Issues

### Issue: Slow Weight Sync

**Symptoms**: Long pauses between training and generation

**Solutions**:

1. **Enable RDMA** (if available):
```bash
export TORCHSTORE_USE_RDMA=1
```

2. **Reduce sync frequency**:
```yaml
training:
  sync_interval: 10  # Sync every 10 steps
```

3. **Use colocated services**:
```yaml
services:
  generator:
    colocate_with: trainer
```

### Issue: Weight Sync Failures

**Symptoms**: Errors in weight transfer, stale weights

**Solutions**:

1. **Check network connectivity**:
```bash
ping other_node
```

2. **Increase timeout**:
```yaml
services:
  weight_sync_timeout: 600  # 10 minutes
```

3. **Enable sync verification**:
```yaml
training:
  verify_weight_sync: true
```

## Training Stability Issues

### Issue: Policy Collapse

**Symptoms**: Entropy drops to zero, reward stops improving

**Solutions**:

1. **Increase KL penalty**:
```yaml
grpo:
  beta: 0.2  # Increase from 0.1
```

2. **Add entropy bonus**:
```yaml
training:
  entropy_coef: 0.01
```

3. **Reduce learning rate**:
```yaml
training:
  learning_rate: 5e-7  # Reduce from 1e-6
```

### Issue: Loss Spikes

**Symptoms**: Sudden loss increases, training instability

**Solutions**:

1. **Enable gradient clipping**:
```yaml
training:
  max_grad_norm: 1.0
```

2. **Reduce clip range**:
```yaml
grpo:
  clip_low: 0.1   # Reduce from 0.2
  clip_high: 0.18 # Reduce from 0.28
```

3. **Use learning rate warmup**:
```yaml
training:
  warmup_steps: 100
```

### Issue: Divergent Training

**Symptoms**: Loss becomes NaN, model outputs garbage

**Solutions**:

1. **Check for data issues**:
```python
# Verify no empty sequences
for batch in dataset:
    assert batch.input_ids.numel() > 0
```

2. **Use BF16 instead of FP16**:
```yaml
training:
  dtype: bfloat16
```

3. **Reduce learning rate significantly**:
```yaml
training:
  learning_rate: 1e-7
```

## Service Issues

### Issue: Service Startup Failures

**Symptoms**: Services fail to initialize

**Solutions**:

1. **Check resource availability**:
```bash
nvidia-smi  # Verify GPU availability
```

2. **Increase startup timeout**:
```yaml
services:
  startup_timeout: 600
```

3. **Check model path**:
```python
from transformers import AutoModelForCausalLM
model = AutoModelForCausalLM.from_pretrained("model_path")  # Verify accessible
```

### Issue: Generator Not Responding

**Symptoms**: Generation hangs, timeouts

**Solutions**:

1. **Check vLLM status**:
```python
# Add health check
await generator.health_check.route()
```

2. **Restart service**:
```python
await generator.restart.fanout()
```

3. **Reduce concurrent requests**:
```yaml
generator:
  max_concurrent_requests: 10
```

## Monarch Issues

### Issue: Monarch Actor Failures

**Symptoms**: Actor crashes, communication errors

**Solutions**:

1. **Enable fault tolerance**:
```yaml
monarch:
  fault_tolerance: true
  max_restarts: 3
```

2. **Increase actor memory**:
```yaml
services:
  actor_memory_mb: 4096
```

3. **Check Monarch logs**:
```bash
export MONARCH_LOG_LEVEL=DEBUG
```

### Issue: Deadlock in Distributed Communication

**Symptoms**: Training hangs, no progress

**Solutions**:

1. **Check for blocking calls**:
```python
# Use async/await correctly
result = await service.method.route(args)  # Correct
# result = service.method.route(args).wait()  # May deadlock
```

2. **Add timeouts**:
```python
result = await asyncio.wait_for(
    service.method.route(args),
    timeout=60.0
)
```

## Installation Issues

### Issue: PyTorch Version Mismatch

**Symptoms**: Import errors, CUDA errors

**Solutions**:

1. **Use provided install script**:
```bash
./scripts/install.sh
```

2. **Verify versions**:
```python
import torch
print(torch.__version__)  # Should be 2.9.0+
```

3. **Clean reinstall**:
```bash
pip uninstall torch torchvision torchaudio
./scripts/install.sh
```

### Issue: Monarch Installation Fails

**Symptoms**: Cannot import monarch

**Solutions**:

1. **Install from source**:
```bash
git clone https://github.com/meta-pytorch/monarch
cd monarch && pip install -e .
```

2. **Check CUDA compatibility**:
```bash
nvcc --version  # Should match PyTorch CUDA
```

## Debugging Tips

### Enable Verbose Logging

```bash
export FORGE_DEBUG=1
export MONARCH_LOG_LEVEL=DEBUG
```

### Profile Services

```python
# Add profiling
with torch.profiler.profile() as prof:
    result = await trainer.train_step.route(batch)
prof.export_chrome_trace("trace.json")
```

### Monitor GPU Utilization

```bash
watch -n 1 nvidia-smi
```

### Test Services Individually

```python
# Test generator
completions = await generator.generate.route(
    prompts=["Hello"],
    max_tokens=10,
)
print(completions[0].text)

# Test trainer
result = await trainer.train_step.route(dummy_batch)
print(result.loss)
```

## Experimental Warning

Both Monarch and torchforge are experimental. Expect:
- API changes between versions
- Incomplete features
- Bugs in edge cases

Check Discord for latest updates and workarounds.

## Resources

- GitHub Issues: https://github.com/meta-pytorch/torchforge/issues
- Discord: https://discord.gg/YsTYBh6PD9
- Monarch Issues: https://github.com/meta-pytorch/monarch/issues
reverse-engineer-anythingmorluto/reaReverse engineer native, managed, Electron/JavaScript, packaged, firmware, and browser targets with REA, and build agent environments (AgentEnv bundles) of web applications from a live instance or a capture, and of Linux desktop apps with a task from a request. Use shipped-artifact or requested runtime evidence to explain features, compare versions, decompile code, or guide a reconstruction. Skip REA for ordinary source-repository architecture analysis.Stars: 472kUpdated Claude CodeCodexVerifiedInstall
diagnosing-superpowersobra/superpowersUse when a superpowers session went wrong and your human partner wants to know why — repeated work, ignored plans, stumbles, poor results, a skill that didn't fire, "it took too long", "why is it so expensive", "what is it doing" — or wants to build a bug report for the superpowers maintainers, for the current session or a past one identified by id or path, on any harness.Stars: 322kUpdated Claude CodeCodexVerifiedInstall
dispatching-parallel-agentsobra/superpowersUse when facing 2+ independent tasks that can be worked on without shared state or sequential dependenciesStars: 322kUpdated Claude CodeCodexVerifiedInstall
executing-plansobra/superpowersUse when executing an implementation plan in the current session as the implementer yourself — your human partner chose inline execution, or no subagent tool is availableStars: 322kUpdated Claude CodeCodexVerifiedInstall