Orchestra-Research/AI-Research-SKILLs / 08-distributed-training/pytorch-fsdp2
pytorch-fsdp2
VerifiedAdds PyTorch FSDP2 (fully_shard) to training scripts with correct init, sharding, mixed precision/offload config, and distributed checkpointing. Use when models exceed single-GPU memory or when you need DTensor-based sharding with DeviceMesh.
| File | Kind | Size |
|---|---|---|
| SKILL.md | skill md | 10.7 KB |
| references/pytorch_dcp_async_recipe.md | markdown | 808 B |
| references/pytorch_dcp_overview.md | markdown | 1022 B |
| references/pytorch_dcp_recipe.md | markdown | 1.0 KB |
| references/pytorch_ddp_notes.md | markdown | 461 B |
| references/pytorch_device_mesh_tutorial.md | markdown | 1.2 KB |
| references/pytorch_examples_fsdp2.md | markdown | 742 B |
| references/pytorch_fsdp1_api.md | markdown | 396 B |
| references/pytorch_fsdp2_tutorial.md | markdown | 2.4 KB |
| references/pytorch_fully_shard_api.md | markdown | 2.8 KB |
| references/pytorch_tp_tutorial.md | markdown | 994 B |
| references/ray_train_fsdp2_example.md | markdown | 592 B |
| references/torchtitan_fsdp_notes.md | markdown | 666 B |
references/pytorch_fully_shard_api.md
View on GitHub# Reference: `torch.distributed.fsdp.fully_shard` API (FSDP2)
**Source (official):** PyTorch docs — `torch.distributed.fsdp.fully_shard`
https://docs.pytorch.org/docs/stable/distributed.fsdp.fully_shard.html
Created: Dec 04, 2024 • Last updated: Oct 13, 2025
## Key facts (paraphrased from the API docs)
### User contract highlights
- `fully_shard(model)` converts `model.parameters()` to **DTensor** at init, then hooks **all-gather** before forward/backward and **free/reshard** after.
- The optimizer **must be initialized with DTensor parameters** and step must happen on DTensors.
- Call `model(input)` (not `model.forward(input)`) so hooks run; otherwise explicitly `unshard()` or register the forward method for hooking.
- Apply `fully_shard` **bottom-up**: shard submodules first, then the root module, to form efficient communication groups and enable overlap.
- `fully_shard` “unions” the module type in-place with `FSDPModule`, enabling methods like `unshard()` / `reshard()`.
> Short excerpt (<= 25 words): “Users generally should not call fully_shard() only on the topmost root module.”
### Signature & core args
`fully_shard(module, *, mesh=None, reshard_after_forward=None, shard_placement_fn=None, mp_policy=MixedPrecisionPolicy(...), offload_policy=OffloadPolicy(), ignored_params=None)`
- **mesh** (`DeviceMesh`):
- 1D mesh ⇒ “classic” FSDP sharding, placement `(Shard(0),)`
- 2D mesh ⇒ Hybrid sharding (HSDP): sharded across one dim, replicated across the other, placement `(Replicate(), Shard(0))`
- **reshard_after_forward**:
- `True`: free unsharded params after forward (re-all-gather during backward)
- `False`: keep unsharded params after forward (avoid backward all-gather)
- `None`: defaults to `True` for non-root, `False` for root
- `int`: reshard to a smaller world-size after forward (must divide shard-dim size)
- **shard_placement_fn**: override per-parameter sharding dim (requires even sharding if not dim-0)
- **ignored_params**: parameters not sharded / not moved / not reduced
## Mixed precision & offload policy classes (same doc page)
### `MixedPrecisionPolicy`
Controls:
- `param_dtype`: dtype used for unsharded parameters during forward/backward
- `reduce_dtype`: dtype used for gradient reduction
- `output_dtype`: dtype used for forward output
- `cast_forward_inputs`: whether to cast forward inputs to `param_dtype`
### `OffloadPolicy` and `CPUOffloadPolicy`
OffloadPolicy controls:
- `param_device` / `reduce_device` / `output_device` (and for CPU offload policy, also `optimizer_state_device`)
## Practical implications for agents
- **Bottom-up sharding** is not optional: it affects grouping and memory/perf.
- **Don’t bypass hooks**: using `model.forward` directly breaks all-gather scheduling.
- **Optimizer construction order matters**: construct optimizer after `fully_shard`.
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| slack-gif-creatoranthropics/skillsKnowledge and utilities for creating animated GIFs optimized for Slack. Provides constraints, validation tools, and animation concepts. Use when users request animated GIFs for Slack like "make me a GIF of X doing Y for Slack." | Stars: 198k | Updated | Claude CodeCodex | Verified | Install |
| ponytailDietrichGebert/ponytailLazy senior dev mode for any coding task: the smallest change that fully solves it, and a reply a busy human understands in one read. | Stars: 194k | Updated | Claude CodeCodex | Verified | Install |
| ponytail-debtDietrichGebert/ponytailHarvest every shortcut: comment into one debt ledger, so deferrals get tracked instead of forgotten. One-shot report. | Stars: 194k | Updated | Claude CodeCodex | Verified | Install |
| ponytail-debtDietrichGebert/ponytailList every `shortcut:` comment (and older `ponytail:` ones) as a debt ledger. One-shot report, changes nothing. Use for "ponytail debt", "what did ponytail defer", "list the shortcuts", /ponytail-debt. | Stars: 194k | Updated | Claude CodeCodex | Verified | Install |