Skip to content

Orchestra-Research/AI-Research-SKILLs / 08-distributed-training/pytorch-fsdp2

pytorch-fsdp2

Verified

Adds PyTorch FSDP2 (fully_shard) to training scripts with correct init, sharding, mixed precision/offload config, and distributed checkpointing. Use when models exceed single-GPU memory or when you need DTensor-based sharding with DeviceMesh.

references/pytorch_dcp_overview.md

View on GitHub
# Reference: Distributed Checkpoint (DCP) overview (torch.distributed.checkpoint)

**Source (official):** PyTorch docs — `torch.distributed.checkpoint`  
https://docs.pytorch.org/docs/stable/distributed.checkpoint.html  
Created: Nov 16, 2022 • Last updated: Oct 08, 2025

## What DCP does
- Supports saving/loading from **multiple ranks in parallel**
- Handles **load-time resharding**, enabling saving with one cluster topology and loading into another
- Produces **multiple files per checkpoint** (often at least one per rank)
- Operates “in place”: the model allocates storage first; DCP loads into that storage

## Important caveats
- The docs warn: **no guarantees of backwards compatibility** across PyTorch versions for saved `state_dict`s.
- Process-group usage: if you pass a process group, only those ranks should call save/load, and all tensors must belong to that group.

## Where to learn usage
The doc links to official “Getting Started with DCP” and “Asynchronous Saving with DCP” recipes.
slack-gif-creatoranthropics/skillsKnowledge and utilities for creating animated GIFs optimized for Slack. Provides constraints, validation tools, and animation concepts. Use when users request animated GIFs for Slack like "make me a GIF of X doing Y for Slack."Stars: 198kUpdated Claude CodeCodexVerifiedInstall
ponytailDietrichGebert/ponytailLazy senior dev mode for any coding task: the smallest change that fully solves it, and a reply a busy human understands in one read.Stars: 194kUpdated Claude CodeCodexVerifiedInstall
ponytail-debtDietrichGebert/ponytailHarvest every shortcut: comment into one debt ledger, so deferrals get tracked instead of forgotten. One-shot report.Stars: 194kUpdated Claude CodeCodexVerifiedInstall
ponytail-debtDietrichGebert/ponytailList every `shortcut:` comment (and older `ponytail:` ones) as a debt ledger. One-shot report, changes nothing. Use for "ponytail debt", "what did ponytail defer", "list the shortcuts", /ponytail-debt.Stars: 194kUpdated Claude CodeCodexVerifiedInstall