Skip to content

Orchestra-Research/AI-Research-SKILLs / 08-distributed-training/pytorch-fsdp2

pytorch-fsdp2

Verified

Adds PyTorch FSDP2 (fully_shard) to training scripts with correct init, sharding, mixed precision/offload config, and distributed checkpointing. Use when models exceed single-GPU memory or when you need DTensor-based sharding with DeviceMesh.

references/pytorch_dcp_recipe.md

View on GitHub
# Reference: Getting Started with Distributed Checkpoint (DCP) recipe

**Source (official):** PyTorch Tutorials recipe — “Getting Started with Distributed Checkpoint (DCP)”  
https://docs.pytorch.org/tutorials/recipes/distributed_checkpoint_recipe.html  
Created: Oct 02, 2023 • Last updated: Jul 10, 2025 • Last verified: Nov 05, 2024

## Key ideas shown in the recipe
- DCP saves/loads in parallel, and supports resharding across topologies at load time.
- It provides helpers under `torch.distributed.checkpoint.state_dict` to manage distributed `state_dict` generation/loading.

## Example structure (high level)
- Wrap application state in a `Stateful` object, so DCP automatically calls `state_dict()` / `load_state_dict()`
- Use `dcp.save(...)` / `dcp.load(...)`
- Use `get_state_dict` / `set_state_dict` helpers to correctly obtain and apply model/optimizer state dicts in distributed settings

## Practical agent guidance
If adding checkpointing to an FSDP2 training script, this recipe’s patterns are the safest default.
slack-gif-creatoranthropics/skillsKnowledge and utilities for creating animated GIFs optimized for Slack. Provides constraints, validation tools, and animation concepts. Use when users request animated GIFs for Slack like "make me a GIF of X doing Y for Slack."Stars: 198kUpdated Claude CodeCodexVerifiedInstall
ponytailDietrichGebert/ponytailLazy senior dev mode for any coding task: the smallest change that fully solves it, and a reply a busy human understands in one read.Stars: 196kUpdated Claude CodeCodexVerifiedInstall
ponytail-debtDietrichGebert/ponytailHarvest every shortcut: comment into one debt ledger, so deferrals get tracked instead of forgotten. One-shot report.Stars: 196kUpdated Claude CodeCodexVerifiedInstall
ponytail-debtDietrichGebert/ponytailList every `shortcut:` comment (and older `ponytail:` ones) as a debt ledger. One-shot report, changes nothing. Use for "ponytail debt", "what did ponytail defer", "list the shortcuts", /ponytail-debt.Stars: 196kUpdated Claude CodeCodexVerifiedInstall