Orchestra-Research/AI-Research-SKILLs / 18-multimodal/openpi
fine-tuning-serving-openpi
VerifiedFine-tune and serve Physical Intelligence OpenPI models (pi0, pi0-fast, pi0.5) using JAX or PyTorch backends for robot policy inference across ALOHA, DROID, and LIBERO environments. Use when adapting pi0 models to custom datasets, converting JAX checkpoints to PyTorch, running policy inference servers, or debugging norm stats and GPU memory issues.
| File | Kind | Size |
|---|---|---|
| SKILL.md | skill md | 13.9 KB |
| references/checkpoints-and-env-map.md | markdown | 2.5 KB |
| references/config-recipes.md | markdown | 1.7 KB |
| references/pytorch-gotchas.md | markdown | 1.4 KB |
| references/remote-client-pattern.md | markdown | 1.2 KB |
| references/training-debugging.md | markdown | 1.2 KB |
references/pytorch-gotchas.md
View on GitHub# PyTorch Precision and Patching Gotchas
## Transformer patch requirement
OpenPI PyTorch requires custom patches applied to the installed `transformers` package. Training or inference without the patch produces subtle incompatibilities.
**Apply patches:**
```bash
cp -r ./src/openpi/models_pytorch/transformers_replace/* \
.venv/lib/python3.11/site-packages/transformers/
```
**Verify the patch is active:**
Check that modified files in the transformers package directory have recent timestamps matching the patch application.
## Patch survives reinstall
If `uv sync` or `pip install` reinstalls `transformers`, the patch is overwritten.
Fix: reapply patches after any dependency reinstall, or run:
```bash
uv cache clean transformers
```
Then reapply the patch.
## OOM while loading checkpoints
Set memory allocation strategy before loading large models:
```bash
export PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True
```
## Resume mode
- `--resume` requires `--exp_name` to match the prior run exactly.
- At least one numeric checkpoint directory must exist under `checkpoints/<config_name>/<exp_name>/`.
- Do not combine `--resume` with other conflicting flags.
## Precision notes
- Default training precision follows the model config.
- When converting from JAX, ensure the output precision matches expectations (bf16 vs fp32).
- Mixed precision settings in PyTorch should align with the source JAX checkpoint precision.
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| receiving-code-reviewobra/superpowersUse when receiving code review feedback, before implementing suggestions, especially if feedback seems unclear or technically questionable - requires technical rigor and verification, not performative agreement or blind implementation | Stars: 323k | Updated | Claude CodeCodex | Verified | Install |
| requesting-code-reviewobra/superpowersUse when completing tasks, implementing major features, or before merging to verify work meets requirements | Stars: 323k | Updated | Claude CodeCodex | Verified | Install |
| using-git-worktreesobra/superpowersUse when starting feature work that needs isolation from current workspace or before executing implementation plans - ensures an isolated workspace exists via native tools or git worktree fallback | Stars: 323k | Updated | Claude CodeCodex | Verified | Install |
| verification-before-completionobra/superpowersUse when about to claim work is complete, fixed, or passing, before committing or creating PRs - requires running verification commands and confirming output before making any success claims; evidence before assertions always | Stars: 323k | Updated | Claude CodeCodex | Verified | Install |