Skip to content

Orchestra-Research/AI-Research-SKILLs / 08-distributed-training/megatron-core

training-llms-megatron

Verified

Trains large language models (2B-462B parameters) using NVIDIA Megatron-Core with advanced parallelism strategies. Use when training models >1B parameters, need maximum GPU efficiency (47% MFU on H100), or require tensor/pipeline/sequence/context/expert parallelism. Production-ready framework used for Nemotron, LLaMA, DeepSeek.

FileKindSize
SKILL.mdskill md9.5 KB
references/benchmarks.mdmarkdown7.2 KB
references/parallelism-guide.mdmarkdown9.4 KB
references/production-examples.mdmarkdown10.6 KB
references/training-recipes.mdmarkdown11.3 KB

references/parallelism-guide.md

View on GitHub
# Parallelism Strategies Guide

Complete guide to parallelism in Megatron-Core: when to use each strategy, performance characteristics, and configuration examples.

## Parallelism Types

### 1. Data Parallel (DP)

**How it works**: Replicate full model on each GPU, split data batches, synchronize gradients.

**Memory**: Low efficiency - full model replication
**Communication**: Low - only gradients (all-reduce)
**Scalability**: Good for small models

**When to use**:
- Models <1B parameters that fit on single GPU
- Plenty of GPUs available
- Limited model complexity

**Configuration**:
```bash
# 8 GPUs, no model parallelism
torchrun --nproc_per_node=8 train.py \
  --tensor-model-parallel-size 1 \
  --pipeline-model-parallel-size 1
# Implicit DP = 8
```

**Performance**:
- Near-linear scaling for small models
- 7B model on 8×A100: ~90% efficiency

### 2. Tensor Parallel (TP)

**How it works**: Split individual layers/tensors across GPUs (column/row partitioning of weight matrices).

**Memory**: Excellent - 1/N reduction per GPU
**Communication**: Very high - all-reduce after every layer
**Scalability**: Best ≤8 GPUs within single node (needs NVLink)

**When to use**:
- Models >10B parameters
- Have NVLink-connected GPUs
- Within single node (network latency kills performance across nodes)

**Configuration**:
```bash
# Split model across 4 GPUs with TP
torchrun --nproc_per_node=4 train.py \
  --tensor-model-parallel-size 4
```

**Performance**:
- **1 node (8 GPUs, NVLink)**: 85-95% efficiency
- **Across nodes**: <50% efficiency (avoid)

**Memory savings**:
```
LLaMA 70B without TP: 140GB (won't fit on 80GB GPU)
LLaMA 70B with TP=4: 35GB per GPU (fits easily)
```

**Communication volume** (70B model):
- Per layer: ~20GB all-reduce
- 80 layers × 20GB = 1.6TB total traffic
- With NVLink (600GB/s): Manageable
- With Ethernet (100Gb/s = 12.5GB/s): Too slow

### 3. Pipeline Parallel (PP)

**How it works**: Divide model layers into stages, assign stages to different GPUs, process microbatches in pipeline.

**Memory**: Very high - divide layers evenly
**Communication**: Low-medium - only activations between stages
**Scalability**: Good across nodes

**Pipeline Schedules**:

**GPipe** (simple but inefficient):
```
GPU0: F F F F ........ B B B B
GPU1: .... F F F F .... B B B B
GPU2: ........ F F F F B B B B
```
Bubble: 50% idle time

**1F1B** (one-forward-one-backward):
```
GPU0: F F F F B B B B B B B B
GPU1: .. F F F F B B B B B B B B
GPU2: .... F F F F B B B B B B B B
```
Bubble: ~25% idle time

**Interleaved 1F1B** (best):
```
GPU0: F1 F2 F3 F4 B1 B2 B3 B4 ...
GPU1: F1 F2 F3 F4 B1 B2 B3 B4 ...
```
Bubble: 5-10% idle time

**When to use**:
- Models >70B parameters
- Multi-node training
- Limited intra-node bandwidth

**Configuration**:
```bash
# 4-stage pipeline
torchrun --nproc_per_node=8 --nnodes=4 train.py \
  --pipeline-model-parallel-size 4 \
  --num-layers 80 \
  --num-layers-per-virtual-pipeline-stage 2  # Interleaved
```

**Performance**:
- Interleaved schedule: 90-95% efficiency
- Standard 1F1B: 75-85% efficiency

### 4. Sequence Parallel (SP)

**How it works**: Split sequence dimension across tensor-parallel GPUs, reduce activation memory.

**Memory**: Reduces activations by TP factor
**Communication**: Same as TP (already using all-reduce)
**Scalability**: Tied to TP

**When to use**:
- Long sequences (>4K tokens)
- Using TP already
- Activation memory is bottleneck

**Configuration**:
```bash
torchrun --nproc_per_node=8 train.py \
  --tensor-model-parallel-size 4 \
  --sequence-parallel  # Requires TP > 1
```

**Memory savings**:
```
70B model, 4K sequence, TP=4:
Without SP: 48GB activations per GPU
With SP: 12GB activations per GPU
Savings: 75%
```

### 5. Context Parallel (CP)

**How it works**: Split very long sequences across GPUs using Ring Attention.

**Memory**: Reduces KV cache and activations
**Communication**: Medium - ring communication pattern
**Scalability**: Good for >8K sequences

**When to use**:
- Sequences >8K tokens
- Long-context models (>32K)
- KV cache memory bottleneck

**Configuration**:
```bash
torchrun --nproc_per_node=8 train.py \
  --context-parallel-size 2 \
  --seq-length 32768  # 32K tokens
```

**Memory savings** (32K sequence):
```
Without CP: 64GB KV cache
With CP=4: 16GB KV cache per GPU
```

### 6. Expert Parallel (EP)

**How it works**: For MoE models, distribute different experts across GPUs.

**Memory**: Excellent - only store 1/N experts per GPU
**Communication**: Low - only route tokens to experts
**Scalability**: Matches number of experts

**When to use**:
- Mixture of Experts models
- Want model capacity without memory cost
- Have ≥8 GPUs

**Configuration**:
```bash
# Mixtral 8x7B: 8 experts
torchrun --nproc_per_node=8 train.py \
  --expert-model-parallel-size 4 \
  --num-experts 8 \
  --tensor-model-parallel-size 2
```

**Memory** (Mixtral 8×7B):
```
Without EP: 8 experts × 7B = 56GB
With EP=4: 2 experts × 7B = 14GB
Savings: 75%
```

## Combining Parallelism Strategies

### 3D Parallelism (TP + PP + DP)

Standard for large models.

**LLaMA 3 70B on 64 GPUs**:
```bash
TP=4  # Within each node
PP=4  # Across nodes
DP=4  # Remaining dimension
Total = 4 × 4 × 4 = 64 GPUs
```

**Memory per GPU**: 70B / 4 (TP) / 4 (PP) = 4.4B params ≈ 20GB

**Configuration**:
```bash
torchrun --nproc_per_node=8 --nnodes=8 train.py \
  --tensor-model-parallel-size 4 \
  --pipeline-model-parallel-size 4
  # DP is implicit: 64 / (4*4) = 4
```

### 4D Parallelism (TP + PP + DP + CP)

For very large models or long context.

**LLaMA 3 405B on 256 GPUs**:
```bash
TP=8   # Max NVLink
PP=8   # Across nodes
CP=2   # Long sequences
DP=2   # Remaining
Total = 8 × 8 × 2 × 2 = 256 GPUs
```

**Configuration**:
```bash
torchrun --nproc_per_node=8 --nnodes=32 train.py \
  --tensor-model-parallel-size 8 \
  --pipeline-model-parallel-size 8 \
  --context-parallel-size 2
```

### 4D + EP (5D Parallelism)

For sparse MoE models.

**DeepSeek-V3 671B (37B active) on 1024 GPUs**:
```bash
TP=2   # Limited by active params
PP=16  # Many stages
EP=64  # 256 experts / 4 experts per GPU
DP=2   # Small data parallel
Total = 2 × 16 × 64 × 2 = 4096 (uses 1024 in practice)
```

## Decision Guide

### By Model Size

| Model Size | GPUs | Recommended Strategy |
|------------|------|---------------------|
| <1B | 1-8 | DP only |
| 1-10B | 8-16 | TP=2-4 + DP |
| 10-70B | 16-64 | TP=4 + PP=2-4 + DP |
| 70-175B | 64-256 | TP=8 + PP=4-8 + DP |
| 175-500B | 256-1024 | TP=8 + PP=8-16 + CP=2 + DP |
| 500B+ | 1024+ | 4D or 5D (with EP) |

### By Hardware Topology

**Single node (8 GPUs with NVLink)**:
```bash
# Up to 70B
TP=8  # Use all NVLink bandwidth
```

**Multiple nodes (InfiniBand)**:
```bash
# Minimize cross-node communication
TP=8      # Within node only
PP=N      # Across nodes
DP=remaining
```

**Limited network (Ethernet)**:
```bash
# Avoid TP across nodes
TP=1-4    # Within node
PP=many   # PP has low communication
```

### By Sequence Length

| Sequence | Parallelism |
|----------|------------|
| <2K | Standard (TP + PP + DP) |
| 2K-8K | + SP (sequence parallel) |
| 8K-32K | + CP=2 (context parallel) |
| 32K+ | + CP=4-8 |

## Performance Characteristics

### Communication Volume (per iteration)

**Data Parallel**: O(model_size) - all-reduce gradients
**Tensor Parallel**: O(model_size × layers) - all-reduce per layer
**Pipeline Parallel**: O(batch × hidden × layers/stages) - activations only
**Context Parallel**: O(sequence × hidden) - ring communication

### Memory Breakdown (70B model example)

Without parallelism:
```
Model parameters: 140GB (FP16)
Gradients: 140GB
Optimizer states: 280GB (Adam)
Activations: 48GB (batch=1, seq=4K)
Total: 608GB (won't fit!)
```

With TP=4, PP=4, DP=4 (64 GPUs):
```
Parameters: 140GB / 4 / 4 = 8.75GB per GPU
Gradients: 8.75GB per GPU
Optimizer: 17.5GB per GPU
Activations: 48GB / 4 / 4 = 3GB per GPU
Total: ~38GB per GPU (fits on A100 80GB)
```

## Best Practices

1. **Start with TP within single node**
   ```bash
   --tensor-model-parallel-size 8  # Use all NVLink
   ```

2. **Add PP for cross-node scaling**
   ```bash
   --pipeline-model-parallel-size 4
   --num-layers-per-virtual-pipeline-stage 2  # Interleaved
   ```

3. **Enable SP when using TP**
   ```bash
   --sequence-parallel  # Free activation savings
   ```

4. **Use CP for long sequences**
   ```bash
   --context-parallel-size 2  # If seq_len > 8K
   ```

5. **Avoid TP across nodes** (network latency kills performance)

6. **Match TP to GPU topology** (TP=8 for 8-GPU nodes)

7. **Profile first iteration** to check memory and communication:
   ```bash
   --profile  # Enable profiling
   --profile-ranks 0  # Profile first rank only
   ```

## Troubleshooting

**High communication overhead (low MFU)**:
- Reduce TP degree (especially across nodes)
- Increase PP degree instead
- Enable interleaved pipeline schedule

**Out of memory**:
- Increase TP/PP (split model more)
- Enable gradient checkpointing:
  ```bash
  --recompute-granularity full
  --recompute-method block
  ```
- Reduce micro-batch size

**Pipeline bubbles (low GPU util)**:
- Use interleaved schedule:
  ```bash
  --num-layers-per-virtual-pipeline-stage 2
  ```
- Increase number of microbatches:
  ```bash
  --global-batch-size 1024
  --micro-batch-size 1  # More microbatches = smaller bubbles
  ```

**Load imbalance in MoE**:
- Tune load balancing:
  ```bash
  --moe-router-load-balancing-type aux_loss
  --moe-aux-loss-coeff 0.01
  ```
- Increase expert parallel degree:
  ```bash
  --expert-model-parallel-size 8  # More experts per GPU
  ```
writing-skillsobra/superpowersUse when creating new skills, editing existing skills, or verifying skills work before deploymentStars: 322kUpdated Claude CodeCodexVerifiedInstall
xlsxanthropics/skillsUse this skill any time a spreadsheet file is the primary input or output. This means any task where the user wants to: open, read, edit, or fix an existing .xlsx, .xlsm, .xltx, .csv, or .tsv file (e.g., adding columns, computing formulas, formatting, charting, cleaning messy data); create a new spreadsheet from scratch or from other data sources; or convert between tabular file formats. Trigger especially when the user references a spreadsheet file by name or path — even casually (like "the xlsx in my downloads") — and wants something done to it or produced from it. Also trigger for cleaning or restructuring messy tabular data files (malformed rows, misplaced headers, junk data) into proper spreadsheets. The deliverable must be a spreadsheet file. Do NOT trigger when the primary deliverable is a Word document, HTML report, standalone Python script, database pipeline, or Google Sheets API integration, even if tabular data is involved.Stars: 198kUpdated Claude CodeCodexVerifiedInstall
shipping-and-launchaddyosmani/agent-skillsPrepares production launches. Use when preparing to deploy to production, or when asking what needs to be in place before shipping. Use when you need a pre-launch checklist, when setting up monitoring, when planning a staged rollout, or when you need a rollback strategy.Stars: 119kUpdated Claude CodeCodexVerifiedInstall
childhoodthedotmack/claude-memBabysit a shipped release after merge, publication, or deployment. Verify published packages, production behavior, installed plugins, and late CI or user reports until the observation window is clean; use babysit for PR readiness before release.Stars: 118kUpdated Claude CodeCodexVerifiedInstall