Skip to content

alirezarezvani/claude-skills / engineering/spinning-up-deep-rl/skills/spinning-up-deep-rl

spinning-up-deep-rl

Verified

Learn or apply deep reinforcement learning with OpenAI Spinning Up: policy gradients, VPG, PPO, SAC, TD3, DDPG, TRPO. Use whenever an RL agent will not learn, reward stays flat, or the user asks how PPO or SAC works.

chapters/ch12-exercises.md

View on GitHub
# Chapter 12: Exercises

## Core Idea
Two problem sets with opposite purposes: Problem Set 1 makes you write the mathematical
core of three algorithms, and Problem Set 2 makes you **watch RL code fail silently** and
find out why — because that is the skill the essay says decides whether you can do this work.

## Frameworks Introduced
- **Problem Set 1: Basics of Implementation** — you are given everything except the math.
  - **1.1 Gaussian Log-Likelihood.** Write a function taking means, log stds and samples,
    returning the log likelihoods. Auto-checked against a known-good implementation on a
    batch of random inputs. The formula is the one in ch7.
  - **1.2 Policy for PPO.** Implement an MLP diagonal Gaussian policy. Auto-checked by
    running 20 epochs on InvertedPendulum-v2 (3-5 minutes). **Bar for success: average score
    over 500 in the last 5 epochs, or 1000 (the maximum) in the last 5.**
  - **1.3 Computation Graph for TD3.** You are given the entire TD3 algorithm *except* the
    loss functions and the intermediate calculations for them. Find "YOUR CODE HERE".
    No automatic checking. Evaluated on HalfCheetah-v2, InvertedPendulum-v2 and one other
    MuJoCo environment of your choosing, with smaller networks ([128,128]), max episode
    length 150, 10 epochs, roughly 10 minutes. **Anecdotal targets within 10 epochs:
    HalfCheetah over 300, InvertedPendulum maxing out at 150.** `--use_soln` runs Spinning
    Up's TD3 instead of yours.

- **Problem Set 2: Algorithm Failure Modes** — the point is the failure, not the fix.
  - **2.1 Value Function Fitting in TRPO.** Compare `train_v_iters=80` against
    `train_v_iters=0` on Hopper-v2, three seeds each, 250 epochs, 4000 steps per epoch.
    **Result: the difference is substantial. With a trained value function the agent makes
    quick progress; with an untrained one it gets stuck early on.** Few factors affect policy
    gradient performance more drastically than the quality of the value function used for
    advantage estimation.
  - **2.2 Silent Bug in DDPG.** Run DDPG with and without a planted bug, three seeds each,
    six runs, ~10 minutes each; plot and compare. Then, **without looking at DDPG's `core.py`**,
    work out what the bug is.

## Worked Example
**The 2.2 bug, and why it is the most instructive page in the book.**

The correct and bugged actor-critic differ in exactly one thing: whether the Q-function
output is squeezed.

```python
# Correct
def forward(self, obs, act):
    q = self.q(torch.cat([obs, act], dim=-1))
    return torch.squeeze(q, -1)   # Critical to ensure q has right shape.

# Bugged
def forward(self, obs, act):
    return self.q(torch.cat([obs, act], dim=-1))   # shape [batch, 1], not [batch]
```

The TF1 version is the same defect: the correct code squeezes to shape `[batch size]`, the
bugged code leaves shape `[batch size, 1]`.

Why that one missing squeeze destroys learning — look at the DDPG graph:

```python
backup = tf.stop_gradient(r_ph + gamma*(1-d_ph)*q_pi_targ)
pi_loss = -tf.reduce_mean(q_pi)
q_loss  = tf.reduce_mean((q - backup)**2)
```

`r_ph` and `d_ph` have shape `[batch size]`. The backup line was written assuming it adds
tensors of the same shape. But it will also happily add tensors of *different* shapes as long
as they are broadcast-compatible — and `[batch size]` and `[batch size, 1]` are compatible,
with results that are not what you expect:

```
x has shape [5], y has shape [5,1]
x * y  ->  shape [5,5]
x + y  ->  shape [5,5]
```

Adding or multiplying a shape-`[5]` tensor by a shape-`[5,1]` tensor returns a shape-`[5,5]`
tensor. So when the Q-functions are not squeezed, `q_pi_targ` has shape `[batch size, 1]`,
the backup becomes a `[batch, batch]` matrix, and the whole Q-loss is meaningless. **Nothing
raises. Nothing warns. The run completes. The agent just learns worse.**

This is ch10's "broken RL code almost always fails silently" reduced to a single missing
`squeeze`. The **Bonus** question is worth sitting with: *are there any choices of
hyperparameters which would have hidden the effects of the bug?*

## Anti-patterns
- **Looking at `core.py` before attempting 2.2.** The exercise is diagnostic practice; the
  answer is worth less than the search.
- **Broadcast-compatible shapes as an implicit contract.** The lesson generalizes far past
  DDPG: any elementwise op between a `[N]` and an `[N,1]` tensor is a silent bug waiting.
  Assert shapes.
- **Skipping Problem Set 2 because it produces no code.** 2.1 produces no code at all and is
  still one of the most decision-relevant results in the book.

## Key Takeaways
1. Problem Set 1 is math-in-code: Gaussian log-likelihood, a PPO policy, TD3's losses.
2. 1.1 and 1.2 self-check; 1.3 does not, and is evaluated on visible learning progress.
3. 2.1: a badly-fit value function does not degrade a policy gradient agent gracefully — it
   gets stuck early.
4. 2.2: one missing `squeeze` silently corrupts the Bellman backup through broadcasting.
5. The challenges past the problem sets are: reimplement algorithms from scratch (ch10), and
   attempt OpenAI's standing Requests for Research.

## Connects To
- **Ch 7**: the diagonal Gaussian log-likelihood formula that Exercise 1.1 asks for.
- **Ch 9**: why the value function baseline quality drives policy gradient performance (2.1).
- **Ch 17 and Ch 18**: the DDPG graph and the TD3 losses the exercises operate on.
- **Ch 10**: "broken RL code almost always fails silently" — Problem Set 2 is the proof.
reverse-engineer-anythingmorluto/reaReverse engineer native, managed, Electron/JavaScript, packaged, firmware, and browser targets with REA, and build agent environments (AgentEnv bundles) of web applications from a live instance or a capture, and of Linux desktop apps with a task from a request. Use shipped-artifact or requested runtime evidence to explain features, compare versions, decompile code, or guide a reconstruction. Skip REA for ordinary source-repository architecture analysis.Stars: 472kUpdated Claude CodeCodexVerifiedInstall
diagnosing-superpowersobra/superpowersUse when a superpowers session went wrong and your human partner wants to know why — repeated work, ignored plans, stumbles, poor results, a skill that didn't fire, "it took too long", "why is it so expensive", "what is it doing" — or wants to build a bug report for the superpowers maintainers, for the current session or a past one identified by id or path, on any harness.Stars: 323kUpdated Claude CodeCodexVerifiedInstall
dispatching-parallel-agentsobra/superpowersUse when facing 2+ independent tasks that can be worked on without shared state or sequential dependenciesStars: 323kUpdated Claude CodeCodexVerifiedInstall
executing-plansobra/superpowersUse when executing an implementation plan in the current session as the implementer yourself — your human partner chose inline execution, or no subagent tool is availableStars: 323kUpdated Claude CodeCodexVerifiedInstall