Orchestra-Research/AI-Research-SKILLs / 06-post-training/verl
verl-rl-training
VerifiedProvides guidance for training LLMs with reinforcement learning using verl (Volcano Engine RL). Use when implementing RLHF, GRPO, PPO, or other RL algorithms for LLM post-training at scale with flexible infrastructure backends.
| File | Kind | Size |
|---|---|---|
| SKILL.md | skill md | 10.1 KB |
| references/api-reference.md | markdown | 6.8 KB |
| references/troubleshooting.md | markdown | 6.6 KB |
references/api-reference.md
View on GitHub# verl API Reference
## Core Classes
### RayPPOTrainer
The central controller for the training loop. Manages resource allocation and coordinates worker groups.
```python
from verl import RayPPOTrainer
trainer = RayPPOTrainer(
config=config,
resource_pool_manager=resource_manager,
ray_worker_group_cls=RayWorkerGroup,
)
trainer.init_workers()
trainer.fit()
```
### ResourcePoolManager
Manages GPU allocation across different worker groups using Ray PlacementGroups.
```python
from verl.trainer.ppo.resource_pool import ResourcePoolManager
manager = ResourcePoolManager(
resource_pool_spec={
"actor_rollout_ref": {"gpu": 4},
"critic": {"gpu": 2},
}
)
```
### RayWorkerGroup
Abstraction for distributed method execution. Spawns Ray actors and dispatches method calls.
```python
from verl.trainer.ppo.ray_worker_group import RayWorkerGroup
worker_group = RayWorkerGroup(
num_workers=8,
worker_cls=ActorRolloutRefWorker,
resource_pool=pool,
)
```
### ActorRolloutRefWorker
Worker class implementing policy training, generation, and reference model computations. Manages hybrid engine mode switching.
```python
# Typically configured via YAML, not instantiated directly
# See configuration section below
```
### RolloutReplica
Interface for inference backends with implementations for vLLM, SGLang, TensorRT-LLM, and HuggingFace.
```python
from verl.workers.rollout import RolloutReplica
# Backend selection via config
rollout:
name: vllm # or: sglang, hf, tensorrt-llm
```
## Configuration Schema
### PPO Configuration (`verl/trainer/config/ppo_trainer.yaml`)
```yaml
# Data configuration
data:
train_files: /path/to/train.parquet
val_files: /path/to/val.parquet
train_batch_size: 256 # Global batch size of prompts
max_prompt_length: 512
max_response_length: 2048
# Algorithm configuration
algorithm:
adv_estimator: gae # gae, grpo, rloo, reinforce_plus_plus
gamma: 0.99 # Discount factor
lam: 0.95 # GAE lambda
use_kl_in_reward: false # Add KL term to reward
# Actor configuration
actor_rollout_ref:
model:
path: Qwen/Qwen2.5-7B-Instruct
backend: fsdp # fsdp, fsdp2, megatron
actor:
ppo_mini_batch_size: 64 # Mini-batch for actor updates
ppo_epochs: 1 # Number of actor update epochs
clip_ratio: 0.2 # PPO clip range
use_kl_loss: true # Use KL loss in actor
kl_loss_coef: 0.001 # KL loss coefficient
kl_loss_type: low_var # KL divergence calculation method
loss_agg_mode: token-mean # token-mean or sequence-mean
gradient_checkpointing: true
max_grad_norm: 1.0 # Gradient clipping
lr: 1e-6 # Learning rate
rollout:
name: vllm # vllm, sglang, hf
n: 8 # Samples per prompt
temperature: 0.7
top_p: 0.95
log_prob_micro_batch_size: 8
# Critic configuration (PPO only)
critic:
model:
path: Qwen/Qwen2.5-7B-Instruct
ppo_mini_batch_size: 64
ppo_epochs: 1 # Defaults to actor epochs
# Trainer configuration
trainer:
total_epochs: 3
n_gpus_per_node: 8
nnodes: 1
save_freq: 100
experiment_name: my_experiment
async_weight_update: false
```
### GRPO Configuration (`docs/algo/grpo.md`)
```yaml
algorithm:
adv_estimator: grpo # Enable GRPO
gamma: 1.0
lam: 1.0
actor_rollout_ref:
rollout:
n: 8 # Must be > 1 for GRPO
actor:
use_kl_loss: true # Required for GRPO
kl_loss_coef: 0.001
kl_loss_type: low_var # or: k1, k2, k3
loss_agg_mode: token-mean
```
### Multi-Turn Configuration (`verl/trainer/config/rollout/rollout.yaml`)
```yaml
actor_rollout_ref:
rollout:
name: sglang # Required for multi-turn
multi_turn:
enable: true
tool_config_path: /path/to/tools.yaml
interaction_config_path: /path/to/interaction.yaml
```
## Reward Functions
### Built-in Reward Types
```yaml
# Model-based reward
reward_model:
path: OpenRLHF/Llama-3-8b-rm-700k
# Custom function-based reward
custom_reward_function:
path: /path/to/reward.py
name: compute_score # Function name, default: compute_score
```
### Custom Reward Function Signature
```python
# reward.py
def compute_score(responses: list[str], ground_truths: list[str], **kwargs) -> list[float]:
"""
Compute rewards for a batch of responses.
Args:
responses: Generated completions
ground_truths: Expected answers from data
**kwargs: Additional metadata
Returns:
List of reward scores (floats)
"""
rewards = []
for response, gt in zip(responses, ground_truths):
# Your reward logic
score = 1.0 if correct(response, gt) else 0.0
rewards.append(score)
return rewards
```
## Backend-Specific Configuration
### FSDP Configuration
```yaml
actor_rollout_ref:
actor:
strategy: fsdp
fsdp_config:
mixed_precision: bf16
sharding_strategy: FULL_SHARD
offload_policy: false
```
### FSDP2 Configuration
```yaml
actor_rollout_ref:
actor:
strategy: fsdp2
fsdp_config:
offload_policy: true # CPU offloading
reshard_after_forward: true
```
### Megatron Configuration
```yaml
actor_rollout_ref:
model:
backend: megatron
actor:
strategy: megatron
tensor_model_parallel_size: 8
pipeline_model_parallel_size: 2
megatron:
use_mbridge: true # Required for format conversion
```
### vLLM Rollout Configuration
```yaml
actor_rollout_ref:
rollout:
name: vllm
tensor_parallel_size: 2
gpu_memory_utilization: 0.9
max_num_seqs: 256
enforce_eager: false
```
### SGLang Rollout Configuration
```yaml
actor_rollout_ref:
rollout:
name: sglang
tp_size: 2
mem_fraction_static: 0.8
context_length: 8192
```
## Algorithm Reference
| Algorithm | `adv_estimator` | Requires Critic | Best For |
|-----------|-----------------|-----------------|----------|
| PPO | `gae` | Yes | Dense rewards, value estimation |
| GRPO | `grpo` | No | Sparse rewards, math/reasoning |
| RLOO | `rloo` | No | Leave-one-out baseline |
| REINFORCE++ | `reinforce_plus_plus` | No | Variance reduction |
| DAPO | `dapo` | No | Doubly-adaptive optimization |
## Vision-Language Model Support
```yaml
actor_rollout_ref:
model:
path: Qwen/Qwen2.5-VL-7B-Instruct
rollout:
name: vllm
enable_vision: true
max_model_len: 32768
```
## LoRA Configuration
```yaml
actor_rollout_ref:
actor:
lora:
enabled: true
r: 16
alpha: 32
target_modules: ["q_proj", "v_proj", "k_proj", "o_proj"]
dropout: 0.05
```
## Resources
- Documentation: https://verl.readthedocs.io/
- GitHub: https://github.com/volcengine/verl
- Paper: https://arxiv.org/abs/2409.19256 (HybridFlow)
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| writing-skillsobra/superpowersUse when creating new skills, editing existing skills, or verifying skills work before deployment | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |
| xlsxanthropics/skillsUse this skill any time a spreadsheet file is the primary input or output. This means any task where the user wants to: open, read, edit, or fix an existing .xlsx, .xlsm, .xltx, .csv, or .tsv file (e.g., adding columns, computing formulas, formatting, charting, cleaning messy data); create a new spreadsheet from scratch or from other data sources; or convert between tabular file formats. Trigger especially when the user references a spreadsheet file by name or path — even casually (like "the xlsx in my downloads") — and wants something done to it or produced from it. Also trigger for cleaning or restructuring messy tabular data files (malformed rows, misplaced headers, junk data) into proper spreadsheets. The deliverable must be a spreadsheet file. Do NOT trigger when the primary deliverable is a Word document, HTML report, standalone Python script, database pipeline, or Google Sheets API integration, even if tabular data is involved. | Stars: 198k | Updated | Claude CodeCodex | Verified | Install |
| shipping-and-launchaddyosmani/agent-skillsPrepares production launches. Use when preparing to deploy to production, or when asking what needs to be in place before shipping. Use when you need a pre-launch checklist, when setting up monitoring, when planning a staged rollout, or when you need a rollback strategy. | Stars: 119k | Updated | Claude CodeCodex | Verified | Install |
| childhoodthedotmack/claude-memBabysit a shipped release after merge, publication, or deployment. Verify published packages, production behavior, installed plugins, and late CI or user reports until the observation window is clean; use babysit for PR readiness before release. | Stars: 118k | Updated | Claude CodeCodex | Verified | Install |