Skip to content

Orchestra-Research/AI-Research-SKILLs / 08-distributed-training/pytorch-fsdp2

pytorch-fsdp2

Verified

Adds PyTorch FSDP2 (fully_shard) to training scripts with correct init, sharding, mixed precision/offload config, and distributed checkpointing. Use when models exceed single-GPU memory or when you need DTensor-based sharding with DeviceMesh.

references/pytorch_device_mesh_tutorial.md

View on GitHub
# Reference: Getting Started with DeviceMesh (PyTorch tutorial)

**Source (official):** PyTorch Recipes — “Getting Started with DeviceMesh”  
https://docs.pytorch.org/tutorials/recipes/distributed_device_mesh.html  
Created: Jan 24, 2024 • Last updated: Jul 18, 2025 • Last verified: Nov 05, 2024

## What DeviceMesh is (as defined by the tutorial)
DeviceMesh is a higher-level abstraction that **manages ProcessGroups**, making it easier to set up the right communication groups for multi-dimensional parallelism.

The tutorial motivation:
- Without DeviceMesh, users must manually compute rank groupings (replicate/shard groups) and create multiple process groups.
- With DeviceMesh, you describe topology with a shape (e.g., 2D mesh), and slice submeshes by dimension name.

## Why this matters for FSDP2
FSDP2 `fully_shard(..., mesh=...)` takes a `DeviceMesh`:
- 1D mesh: standard full sharding across DP workers.
- 2D mesh: hybrid sharding (HSDP), combining replication + sharding across mesh dimensions.

So the agent should:
- Prefer to create a DeviceMesh early (after init_process_group and setting CUDA device).
- Pass the correct (sub)mesh into `fully_shard` if composing with TP or other dimensions.
slack-gif-creatoranthropics/skillsKnowledge and utilities for creating animated GIFs optimized for Slack. Provides constraints, validation tools, and animation concepts. Use when users request animated GIFs for Slack like "make me a GIF of X doing Y for Slack."Stars: 198kUpdated Claude CodeCodexVerifiedInstall
ponytailDietrichGebert/ponytailLazy senior dev mode for any coding task: the smallest change that fully solves it, and a reply a busy human understands in one read.Stars: 194kUpdated Claude CodeCodexVerifiedInstall
ponytail-debtDietrichGebert/ponytailHarvest every shortcut: comment into one debt ledger, so deferrals get tracked instead of forgotten. One-shot report.Stars: 194kUpdated Claude CodeCodexVerifiedInstall
ponytail-debtDietrichGebert/ponytailList every `shortcut:` comment (and older `ponytail:` ones) as a debt ledger. One-shot report, changes nothing. Use for "ponytail debt", "what did ponytail defer", "list the shortcuts", /ponytail-debt.Stars: 194kUpdated Claude CodeCodexVerifiedInstall