Orchestra-Research/AI-Research-SKILLs / 18-multimodal/openvla-oft
fine-tuning-openvla-oft
VerifiedFine-tunes and evaluates OpenVLA-OFT and OpenVLA-OFT+ policies for robot action generation with continuous action heads, LoRA adaptation, and FiLM conditioning on LIBERO simulation and ALOHA real-world setups. Use when reproducing OpenVLA-OFT paper results, training custom VLA action heads (L1 or diffusion), deploying server-client inference for ALOHA, or debugging normalization, LoRA merge, and cross-GPU issues.
| File | Kind | Size |
|---|---|---|
| SKILL.md | skill md | 14.5 KB |
| references/aloha-workflow.md | markdown | 3.8 KB |
| references/config-troubleshooting.md | markdown | 3.9 KB |
| references/libero-workflow.md | markdown | 3.0 KB |
| references/paper-and-checkpoints.md | markdown | 2.5 KB |
references/aloha-workflow.md
View on GitHub# ALOHA Workflow
## Scope
Use this guide for OpenVLA-OFT+ training and real-robot evaluation with the ALOHA stack.
The ALOHA path uses server-client inference:
- Server machine hosts the VLA model and exposes `/act`.
- Client machine controls robot env and requests actions from the server.
## 1) Prepare environments
Server-side environment:
```bash
conda create -n openvla-oft python=3.10 -y
conda activate openvla-oft
pip3 install torch torchvision torchaudio
pip install -e .
pip install uvicorn fastapi json-numpy
```
Client-side environment:
```bash
conda create -n openvla-oft-aloha python=3.10 -y
conda activate openvla-oft-aloha
pip3 install torch torchvision torchaudio
pip install -e .
pip install -r experiments/robot/aloha/requirements_aloha.txt
```
## 2) Preprocess and split raw demonstrations
```bash
python experiments/robot/aloha/preprocess_split_aloha_data.py \
--dataset_path /path/to/aloha_raw/task_name/ \
--out_base_dir /path/to/aloha_preprocessed/ \
--percent_val 0.05
```
Repeat preprocessing per object/task variant, then convert to unified RLDS dataset using the RLDS builder flow.
RLDS builder reference: https://github.com/moojink/rlds_dataset_builder
## 3) Register dataset and constants
Add dataset entries in:
- `prismatic/vla/datasets/rlds/oxe/configs.py`
- `prismatic/vla/datasets/rlds/oxe/transforms.py`
- `prismatic/vla/datasets/rlds/oxe/mixtures.py`
Set platform constants in `prismatic/vla/constants.py`:
- Set `NUM_ACTIONS_CHUNK` to match control frequency (often 25 for 25 Hz).
- Keep ALOHA normalization type for absolute joint-angle actions (`BOUNDS`).
- Avoid clipping normalization for absolute-angle output.
## 4) Launch OFT+ training
```bash
torchrun --standalone --nnodes 1 --nproc-per-node 8 vla-scripts/finetune.py \
--vla_path openvla/openvla-7b \
--data_root_dir /PATH/TO/RLDS/DATASETS/ \
--dataset_name aloha_task_name \
--run_root_dir /YOUR/CHECKPOINTS/ \
--use_l1_regression True \
--use_diffusion False \
--use_film True \
--num_images_in_input 3 \
--use_proprio True \
--batch_size 4 \
--learning_rate 5e-4 \
--num_steps_before_decay 50000 \
--max_steps 100005 \
--use_val_set True \
--val_freq 10000 \
--save_freq 10000 \
--save_latest_checkpoint_only False \
--image_aug True \
--lora_rank 32 \
--wandb_entity YOUR_WANDB_ENTITY \
--wandb_project YOUR_WANDB_PROJECT
```
High-impact knobs:
- `use_film=True` for language grounding in OFT+.
- `num_images_in_input=3` for high + left wrist + right wrist streams.
- LR decay timing relative to dataset size.
## 5) Deploy VLA server
On GPU server:
```bash
python vla-scripts/deploy.py \
--pretrained_checkpoint /PATH/TO/FINETUNED/CHECKPOINT/ \
--use_l1_regression True \
--use_film True \
--num_images_in_input 3 \
--use_proprio True \
--center_crop True \
--unnorm_key aloha_task_name
```
Notes:
- Default API endpoint: `http://<server-ip>:8777/act`
- Ensure client can resolve `vla_server_url`.
## 6) Run client-side robot evaluation
```bash
python experiments/robot/aloha/run_aloha_eval.py \
--center_crop True \
--num_open_loop_steps 25 \
--use_vla_server True \
--vla_server_url http://<SERVER_IP>:8777 \
--num_rollouts_planned 50 \
--max_steps 1500
```
During rollout:
- Script prompts operator to start.
- Script asks for success label (`y` or `n`) after each rollout.
- Logs and replay videos are saved locally.
## 7) Troubleshooting notes
ROS/libffi import issue on client:
```bash
conda install -c conda-forge libffi
```
Action quality issues:
- Check server and training config parity (`use_film`, `num_images_in_input`, `lora_rank`).
- Check `unnorm_key` against dataset stats.
- Keep `num_open_loop_steps` aligned with trained chunk size.
Cross-device performance drop:
- Merge LoRA on target hardware before final evaluation.
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| writing-skillsobra/superpowersUse when creating new skills, editing existing skills, or verifying skills work before deployment | Stars: 323k | Updated | Claude CodeCodex | Verified | Install |
| xlsxanthropics/skillsUse this skill any time a spreadsheet file is the primary input or output. This means any task where the user wants to: open, read, edit, or fix an existing .xlsx, .xlsm, .xltx, .csv, or .tsv file (e.g., adding columns, computing formulas, formatting, charting, cleaning messy data); create a new spreadsheet from scratch or from other data sources; or convert between tabular file formats. Trigger especially when the user references a spreadsheet file by name or path — even casually (like "the xlsx in my downloads") — and wants something done to it or produced from it. Also trigger for cleaning or restructuring messy tabular data files (malformed rows, misplaced headers, junk data) into proper spreadsheets. The deliverable must be a spreadsheet file. Do NOT trigger when the primary deliverable is a Word document, HTML report, standalone Python script, database pipeline, or Google Sheets API integration, even if tabular data is involved. | Stars: 198k | Updated | Claude CodeCodex | Verified | Install |
| shipping-and-launchaddyosmani/agent-skillsPrepares production launches. Use when preparing to deploy to production, or when asking what needs to be in place before shipping. Use when you need a pre-launch checklist, when setting up monitoring, when planning a staged rollout, or when you need a rollback strategy. | Stars: 119k | Updated | Claude CodeCodex | Verified | Install |
| childhoodthedotmack/claude-memBabysit a shipped release after merge, publication, or deployment. Verify published packages, production behavior, installed plugins, and late CI or user reports until the observation window is clean; use babysit for PR readiness before release. | Stars: 118k | Updated | Claude CodeCodex | Verified | Install |