Skip to content

alirezarezvani/claude-skills / engineering/spinning-up-deep-rl/skills/spinning-up-deep-rl

spinning-up-deep-rl

Verified

Learn or apply deep reinforcement learning with OpenAI Spinning Up: policy gradients, VPG, PPO, SAC, TD3, DDPG, TRPO. Use whenever an RL agent will not learn, reward stays flat, or the user asks how PPO or SAC works.

chapters/ch06-plotting-results.md

View on GitHub
# Chapter 6: Plotting Results

## Core Idea
The plotter reads a directory prefix, autocompletes every matching run beneath it, averages
across seeds by default, and resolves the pseudo-metric `Performance` to the *correct*
performance measure for each algorithm family — so a fair on-policy versus off-policy
comparison is one command.

## Frameworks Introduced
- **`Performance` as a family-aware alias**: `Performance` is not a real output of any
  algorithm. The plotter resolves it per logdir to `AverageEpRet` for on-policy algorithms
  and `AverageTestEpRet` for off-policy ones.
  - When to use: always, unless you specifically want one raw column. It is the default `-y`.
  - Why it matters: on-policy performance is the average return of the batch just collected;
    off-policy performance is measured by separate deterministic test rollouts. Plotting the
    same raw column for both would compare two different quantities.

- **Prefix autocompletion**: logdirs are searched recursively and prefixes expand. Give the
  plotter `data/bench_algo` and it finds `bench_algo1` and `bench_algo2` with all their seeds.
  - How: name experiments with a shared prefix at launch and comparison becomes free.

- **Average by default, `--count` to disaggregate**: by default, y-values are averaged across
  all results sharing an `exp_name` — typically identical experiments differing only in seed.
  `--count` shows each curve separately.

## Key Concepts
- **`--xaxis` / `-x`**: which column is the x-axis. Default `TotalEnvInteracts`, i.e. sample
  efficiency is the default framing, not wall-clock or epochs.
- **`--value` / `-y`**: which columns to graph; multiple values produce multiple graphs.
- **`--smooth S`**: average over a fixed window of width S. The book's own benchmark plots
  use a window of 11 epochs (ch13).
- **`--legend` / `-l`**: overrides the automatic legend, which uses `exp_name` from
  `config.json`. One string per *matched* directory, which may exceed the number of logdir
  arguments you passed.
- **`--select` / `--exclude`**: keep or drop curves whose logdir contains all of / any of
  these substrings. The way to narrow an over-eager autocomplete.

## Code Examples
```bash
python -m spinup.run plot [path/to/output_directory ...] [--legend [LEGEND ...]] \
    [--xaxis XAXIS] [--value [VALUE ...]] [--count] [--smooth S] \
    [--select [SEL ...]] [--exclude [EXC ...]]
```

Comparing two algorithms across all their seeds, relying on autocompletion:

```
data/
    bench_algo1/
        bench_algo1-seed0/
        bench_algo1-seed10/
    bench_algo2/
        bench_algo2-seed0/
        bench_algo2-seed10/
```

```bash
python spinup/utils/plot.py data/bench_algo
```

## Anti-patterns
- **Passing one legend string per logdir argument.** Autocompletion may match several
  directories per argument; you need one legend entry per match, unless you narrowed the
  set with `--select` / `--exclude`.
- **Plotting `AverageEpRet` for an off-policy algorithm** and calling it performance — that
  is the behavior policy with exploration noise, not the evaluated policy. Use `Performance`.
- **Heavy smoothing to make a result look clean.** Smoothing is a display parameter; it does
  not change what the seeds did, and ch10 asks you to show seed variance rather than hide it.

## Key Takeaways
1. `Performance` is the correct default because it is algorithm-family aware.
2. Prefix autocompletion plus shared `exp_name` prefixes is the intended comparison workflow.
3. Seed averaging is on by default; `--count` reveals the individual curves that average hides.
4. The default x-axis is environment interactions — sample efficiency is the default question.

## Connects To
- **Ch 5**: `progress.txt`, the plotter's input.
- **Ch 4**: the suffix rules that group seeds into one folder for averaging.
- **Ch 13**: the benchmark plots, produced with 10 seeds and an 11-epoch smoothing window.
reverse-engineer-anythingmorluto/reaReverse engineer native, managed, Electron/JavaScript, packaged, firmware, and browser targets with REA, and build agent environments (AgentEnv bundles) of web applications from a live instance or a capture, and of Linux desktop apps with a task from a request. Use shipped-artifact or requested runtime evidence to explain features, compare versions, decompile code, or guide a reconstruction. Skip REA for ordinary source-repository architecture analysis.Stars: 472kUpdated Claude CodeCodexUnverifiedInstall
diagnosing-superpowersobra/superpowersUse when a superpowers session went wrong and your human partner wants to know why — repeated work, ignored plans, stumbles, poor results, a skill that didn't fire, "it took too long", "why is it so expensive", "what is it doing" — or wants to build a bug report for the superpowers maintainers, for the current session or a past one identified by id or path, on any harness.Stars: 322kUpdated Claude CodeCodexVerifiedInstall
dispatching-parallel-agentsobra/superpowersUse when facing 2+ independent tasks that can be worked on without shared state or sequential dependenciesStars: 322kUpdated Claude CodeCodexVerifiedInstall
executing-plansobra/superpowersUse when executing an implementation plan in the current session as the implementer yourself — your human partner chose inline execution, or no subagent tool is availableStars: 322kUpdated Claude CodeCodexVerifiedInstall