Orchestra-Research/AI-Research-SKILLs / 08-distributed-training/pytorch-fsdp2
pytorch-fsdp2
VerifiedAdds PyTorch FSDP2 (fully_shard) to training scripts with correct init, sharding, mixed precision/offload config, and distributed checkpointing. Use when models exceed single-GPU memory or when you need DTensor-based sharding with DeviceMesh.
| File | Kind | Size |
|---|---|---|
| SKILL.md | skill md | 10.7 KB |
| references/pytorch_dcp_async_recipe.md | markdown | 808 B |
| references/pytorch_dcp_overview.md | markdown | 1022 B |
| references/pytorch_dcp_recipe.md | markdown | 1.0 KB |
| references/pytorch_ddp_notes.md | markdown | 461 B |
| references/pytorch_device_mesh_tutorial.md | markdown | 1.2 KB |
| references/pytorch_examples_fsdp2.md | markdown | 742 B |
| references/pytorch_fsdp1_api.md | markdown | 396 B |
| references/pytorch_fsdp2_tutorial.md | markdown | 2.4 KB |
| references/pytorch_fully_shard_api.md | markdown | 2.8 KB |
| references/pytorch_tp_tutorial.md | markdown | 994 B |
| references/ray_train_fsdp2_example.md | markdown | 592 B |
| references/torchtitan_fsdp_notes.md | markdown | 666 B |
references/pytorch_fsdp2_tutorial.md
View on GitHub# Reference: Getting Started with Fully Sharded Data Parallel (FSDP2) tutorial
**Source (official):** PyTorch Tutorials — “Getting Started with Fully Sharded Data Parallel (FSDP2)”
https://docs.pytorch.org/tutorials/intermediate/FSDP_tutorial.html
Created: Mar 17, 2022 • Last updated: Sep 02, 2025 • Last verified: Nov 05, 2024
## What the tutorial emphasizes
### How FSDP2 differs from DDP and FSDP1
- FSDP shards **parameters, gradients, and optimizer state**; parameters are all-gathered for compute and reduce-scattered for grads.
- Compared to FSDP1, FSDP2:
- uses **DTensor per-parameter sharding** (more direct manipulation; sharded state dicts)
- improves memory management for more deterministic memory behavior
- supports extensibility points for custom all-gather (e.g., float8/NF4 use cases)
### Model initialization flow (meta-device pattern)
The tutorial’s migration section shows a typical pattern:
- initialize model on `meta`
- apply `fully_shard` to the intended layers (policy expressed by explicit calls)
- apply `fully_shard` to the root module
- materialize weights via `to_empty(device="cuda")`, then run `reset_parameters()`
### State dict workflows
The tutorial describes two main ways:
**A) DTensor APIs (manual)**
- Loading: use `distribute_tensor(full_tensor, meta_param.device_mesh, meta_param.placements)` then `model.load_state_dict(..., assign=True)`
- Saving: call `DTensor.full_tensor()` to all-gather; optionally CPU-offload on rank0 to avoid peak GPU memory
**B) DCP distributed state-dict helpers (recommended when no custom handling needed)**
- Loading: `set_model_state_dict(..., StateDictOptions(full_state_dict=True, broadcast_from_rank0=True))`
- Saving: `get_model_state_dict(..., StateDictOptions(full_state_dict=True, cpu_offload=True))`
- Points to `pytorch/examples` for optimizer state dict save/load with `set_optimizer_state_dict` / `get_optimizer_state_dict`
### Migration guide mapping
The tutorial explicitly maps FSDP1 concepts to FSDP2:
- `sharding_strategy` ↔ `reshard_after_forward` (+ 2D mesh for HYBRID)
- `cpu_offload` ↔ `offload_policy` (`CPUOffloadPolicy`)
- `no_sync()` ↔ `set_requires_gradient_sync`
- `sync_module_states` moves to DCP broadcast-from-rank0 flows
## Practical takeaways for agents
- Express wrapping policy by **explicitly applying `fully_shard`** to chosen submodules.
- Use DCP APIs for flexible checkpointing and resharding unless you must interop with third-party formats.
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| slack-gif-creatoranthropics/skillsKnowledge and utilities for creating animated GIFs optimized for Slack. Provides constraints, validation tools, and animation concepts. Use when users request animated GIFs for Slack like "make me a GIF of X doing Y for Slack." | Stars: 198k | Updated | Claude CodeCodex | Verified | Install |
| ponytailDietrichGebert/ponytailLazy senior dev mode for any coding task: the smallest change that fully solves it, and a reply a busy human understands in one read. | Stars: 194k | Updated | Claude CodeCodex | Verified | Install |
| ponytail-debtDietrichGebert/ponytailHarvest every shortcut: comment into one debt ledger, so deferrals get tracked instead of forgotten. One-shot report. | Stars: 194k | Updated | Claude CodeCodex | Verified | Install |
| ponytail-debtDietrichGebert/ponytailList every `shortcut:` comment (and older `ponytail:` ones) as a debt ledger. One-shot report, changes nothing. Use for "ponytail debt", "what did ponytail defer", "list the shortcuts", /ponytail-debt. | Stars: 194k | Updated | Claude CodeCodex | Verified | Install |