alirezarezvani/claude-skills / engineering/spinning-up-deep-rl/skills/spinning-up-deep-rl
spinning-up-deep-rl
VerifiedLearn or apply deep reinforcement learning with OpenAI Spinning Up: policy gradients, VPG, PPO, SAC, TD3, DDPG, TRPO. Use whenever an RL agent will not learn, reward stays flat, or the user asks how PPO or SAC works.
chapters/ch04-running-experiments.md
View on GitHub# Chapter 4: Running Experiments
## Core Idea
`spinup/run.py` turns every keyword argument of every algorithm into a command-line flag,
and turns a list of values for a flag into a series of experiments — so hyperparameter
sweeps and multi-seed runs are one command, not a script.
## Frameworks Introduced
- **One flag per kwarg**: if `kwarg` is a valid keyword argument of an algorithm function,
`--kwarg` sets it. `python -m spinup.run [algo] --help` prints the docstring readout.
- How: `python -m spinup.run [algo name] [experiment flags]`.
- **Multiple values means multiple experiments**: providing more than one value for an
argument launches one experiment per combination, **in series**. They do not run in
parallel because a single experiment already soaks up enough resources that concurrency
buys no speedup.
- When to use: seed sweeps (the ch10 rigor requirement) and hyperparameter ablations.
- **ExperimentGrid**: the in-script equivalent, based on but simpler than rllab's
VariantGenerator. `spinup.run` uses one under the hood.
- How: `eg.add(param_name, values, shorthand, in_name)` then `eg.run(thunk, **run_kwargs)`.
`in_name` forces a parameter into the experiment name even when it does not vary.
- Difference from the CLI: no shortcut kwargs — you must write `ac_kwargs:hidden_sizes`,
not `hid`.
- **Shorthand-driven save directories**: results land in
`data_dir/[outer_prefix]exp_name[suffix]/[inner_prefix]exp_name[suffix]_s[seed]`.
The suffix encodes only the hyperparameters that *differ across the launched experiments*,
and never the seed — so runs that differ only by seed group into the same folder, which is
exactly what the plotter needs to average over seeds.
## Key Concepts
- **`--env` / `--env_name`**: a Gym environment name, converted internally into the `env_fn`
callable every algorithm actually takes.
- **`--hid` / `--act`**: shortcut flags for `ac_kwargs:hidden_sizes` and `ac_kwargs:activation`,
valid for all current algorithms.
- **`--cpu` / `--num_cpu`**: launch with this many MPI-connected processes; `auto` uses all
available. Raises an error for algorithms that do not support parallelization.
- **`--dt` / `--datestamp`**: put timestamps in the save directory names.
- **User-supplied shorthand**: square brackets after a flag, e.g. `--hid[h]`, control the
directory-name abbreviation. Without one, a shorthand is derived automatically
(`clip_ratio` becomes `cli`).
- **`eval()` passthrough**: flag values pass through `eval()` before use, so you can name
functions and objects directly, e.g. `--act torch.nn.ELU`.
- **Dict kwargs**: `--key:v1 value_1 --key:v2 value_2` instead of `--key dict(v1=..., v2=...)`.
## Code Examples
The quickstart, with every flag doing something distinct:
```bash
python -m spinup.run ppo --exp_name ppo_ant --env Ant-v2 --clip_ratio 0.1 0.2 \
--hid[h] [32,32] [64,32] --act torch.nn.Tanh --seed 0 10 20 --dt \
--data_dir path/to/data
```
Choosing the backend explicitly:
```bash
python -m spinup.run ppo_pytorch --env Walker2d-v2 --exp_name walker
python -m spinup.run ppo_tf1 --env Walker2d-v2 --exp_name walker
# bare `ppo` reads spinup/user_config.py for the default backend
```
From a script:
```python
from spinup import ppo_pytorch as ppo
import gym
env_fn = lambda: gym.make('LunarLander-v2')
ac_kwargs = dict(hidden_sizes=[64, 64])
logger_kwargs = dict(output_dir='path/to/output_dir', exp_name='experiment_name')
ppo(env_fn=env_fn, ac_kwargs=ac_kwargs, steps_per_epoch=5000, epochs=250,
logger_kwargs=logger_kwargs)
```
ExperimentGrid, from `spinup/examples/pytorch/bench_ppo_cartpole.py`:
```python
from spinup.utils.run_utils import ExperimentGrid
from spinup import ppo_pytorch
import torch
eg = ExperimentGrid(name='ppo-pyt-bench')
eg.add('env_name', 'CartPole-v0', '', True)
eg.add('seed', [10*i for i in range(args.num_runs)])
eg.add('epochs', 10)
eg.add('steps_per_epoch', 4000)
eg.add('ac_kwargs:hidden_sizes', [(32,), (64,64)], 'hid')
eg.add('ac_kwargs:activation', [torch.nn.Tanh, torch.nn.ReLU], '')
eg.run(ppo_pytorch, num_cpu=args.cpu)
```
## Worked Example
`python -m spinup.run ddpg_tf1 --env Hopper-v2 --hid[h] [300] [128,128] --act tf.nn.tanh tf.nn.relu`
Two `hid` values times two `act` values is four experiments, run in series, producing four
suffixes:
```
_h128-128_ac-actrelu
_h128-128_ac-acttanh
_h300_ac-actrelu
_h300_ac-acttanh
```
`h` came from the user-supplied `[h]`; `ac-act` was derived automatically from the true flag
name `ac_kwargs:activation`. Add `--seed 0 10 20` and you get twelve runs in four folders —
three seeds grouped per configuration, which is the grouping the plotter averages over.
## Anti-patterns
- **Running the per-algorithm files directly** (`spinup/algos/BACKEND/ALGO/ALGO.py`). The
command-line support there is vestigial, takes a different argument set, and is explicitly
not the recommended way to run experiments.
- **Using ZShell without escaping square brackets**: ZShell treats them as special characters
and Spinning Up uses them for both list values and shorthands.
- **One seed.** RL algorithms have high variance; the flag exists so you use it (ch10 asks
for at least 3, ideally 10 or more).
- **Expecting parallel experiment launches.** Sweeps run in series by design.
## Key Takeaways
1. Every kwarg is a flag; every list of flag values is a sweep.
2. Sweeps run in series; MPI parallelism is per-experiment via `--cpu`, and only for
algorithms that support it.
3. Save-directory suffixes encode only the varying hyperparameters, never the seed — that
grouping is what makes multi-seed plotting work.
4. `ExperimentGrid` is the scripted form and loses only the shortcut kwargs.
5. Values go through `eval()`, so activations and other objects can be named on the command line.
## Connects To
- **Ch 5**: what the directories those flags create actually contain.
- **Ch 6**: the plotter, which consumes this directory structure and its autocompletion.
- **Ch 10**: why multi-seed runs are a rigor requirement, not a convenience.
- **Ch 20**: `ExperimentGrid`, `call_experiment` and `setup_logger_kwargs` in Run Utils.
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| reverse-engineer-anythingmorluto/reaReverse engineer native, managed, Electron/JavaScript, packaged, firmware, and browser targets with REA, and build agent environments (AgentEnv bundles) of web applications from a live instance or a capture, and of Linux desktop apps with a task from a request. Use shipped-artifact or requested runtime evidence to explain features, compare versions, decompile code, or guide a reconstruction. Skip REA for ordinary source-repository architecture analysis. | Stars: 472k | Updated | Claude CodeCodex | Verified | Install |
| diagnosing-superpowersobra/superpowersUse when a superpowers session went wrong and your human partner wants to know why — repeated work, ignored plans, stumbles, poor results, a skill that didn't fire, "it took too long", "why is it so expensive", "what is it doing" — or wants to build a bug report for the superpowers maintainers, for the current session or a past one identified by id or path, on any harness. | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |
| dispatching-parallel-agentsobra/superpowersUse when facing 2+ independent tasks that can be worked on without shared state or sequential dependencies | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |
| executing-plansobra/superpowersUse when executing an implementation plan in the current session as the implementer yourself — your human partner chose inline execution, or no subagent tool is available | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |