Orchestra-Research/AI-Research-SKILLs / 08-distributed-training/pytorch-fsdp2
pytorch-fsdp2
VerifiedAdds PyTorch FSDP2 (fully_shard) to training scripts with correct init, sharding, mixed precision/offload config, and distributed checkpointing. Use when models exceed single-GPU memory or when you need DTensor-based sharding with DeviceMesh.
| File | Kind | Size |
|---|---|---|
| SKILL.md | skill md | 10.7 KB |
| references/pytorch_dcp_async_recipe.md | markdown | 808 B |
| references/pytorch_dcp_overview.md | markdown | 1022 B |
| references/pytorch_dcp_recipe.md | markdown | 1.0 KB |
| references/pytorch_ddp_notes.md | markdown | 461 B |
| references/pytorch_device_mesh_tutorial.md | markdown | 1.2 KB |
| references/pytorch_examples_fsdp2.md | markdown | 742 B |
| references/pytorch_fsdp1_api.md | markdown | 396 B |
| references/pytorch_fsdp2_tutorial.md | markdown | 2.4 KB |
| references/pytorch_fully_shard_api.md | markdown | 2.8 KB |
| references/pytorch_tp_tutorial.md | markdown | 994 B |
| references/ray_train_fsdp2_example.md | markdown | 592 B |
| references/torchtitan_fsdp_notes.md | markdown | 666 B |
references/pytorch_tp_tutorial.md
View on GitHub# Reference: Tensor Parallel (TP) tutorial (and how it composes with FSDP)
**Source (official):** PyTorch Tutorials — “Large Scale Transformer model training with Tensor Parallel (TP)”
https://docs.pytorch.org/tutorials/intermediate/TP_tutorial.html
Created: Apr 19, 2024 • Last updated: Jul 18, 2025 • Last verified: Nov 05, 2024
## Key composition pattern: TP intra-host + FSDP inter-host
The tutorial recommends:
- Run TP on a fast intra-host fabric (e.g., NVLink).
- Run FSDP across hosts (inter-host).
It shows a **2D DeviceMesh** pattern and slicing:
- `mesh_2d = init_device_mesh("cuda", (dp, tp))`
- `tp_mesh = mesh_2d["tp"]` and `dp_mesh = mesh_2d["dp"]`
- Apply TP with `parallelize_module(..., tp_mesh, ...)`
- Apply FSDP2 with `fully_shard(..., mesh=dp_mesh, ...)`
## Practical agent guidance
If the user is already doing TP:
- Ensure FSDP2 `mesh` only includes the DP dimension (often inter-host).
- Leave the TP dimension to `torch.distributed.tensor.parallel`.
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| slack-gif-creatoranthropics/skillsKnowledge and utilities for creating animated GIFs optimized for Slack. Provides constraints, validation tools, and animation concepts. Use when users request animated GIFs for Slack like "make me a GIF of X doing Y for Slack." | Stars: 198k | Updated | Claude CodeCodex | Verified | Install |
| ponytailDietrichGebert/ponytailLazy senior dev mode for any coding task: the smallest change that fully solves it, and a reply a busy human understands in one read. | Stars: 194k | Updated | Claude CodeCodex | Verified | Install |
| ponytail-debtDietrichGebert/ponytailHarvest every shortcut: comment into one debt ledger, so deferrals get tracked instead of forgotten. One-shot report. | Stars: 194k | Updated | Claude CodeCodex | Verified | Install |
| ponytail-debtDietrichGebert/ponytailList every `shortcut:` comment (and older `ponytail:` ones) as a debt ledger. One-shot report, changes nothing. Use for "ponytail debt", "what did ponytail defer", "list the shortcuts", /ponytail-debt. | Stars: 194k | Updated | Claude CodeCodex | Verified | Install |