alirezarezvani/claude-skills / engineering/spinning-up-deep-rl/skills/spinning-up-deep-rl
spinning-up-deep-rl
VerifiedLearn or apply deep reinforcement learning with OpenAI Spinning Up: policy gradients, VPG, PPO, SAC, TD3, DDPG, TRPO. Use whenever an RL agent will not learn, reward stays flat, or the user asks how PPO or SAC works.
chapters/ch18-td3.md
View on GitHub# Chapter 18: Twin Delayed DDPG (TD3)
## Core Idea
DDPG's common failure mode is that **the learned Q-function begins to dramatically
overestimate Q-values, which then breaks the policy, because the policy exploits the errors
in the Q-function.** TD3 fixes this with three tricks, and nothing else changes.
## Quick Facts
- **Off-policy.**
- **Continuous action spaces only.**
- The Spinning Up implementation **does not support parallelization.**
## Frameworks Introduced
- **Trick One: Clipped Double-Q Learning.** Learn **two** Q-functions instead of one (hence
"twin") and **use the smaller of the two Q-values to form the targets** in the Bellman error
loss functions. Both are then regressed toward that single shared target:
```
y(r,s',d) = r + gamma*(1-d) * min_{i=1,2} Q_{phi_i,targ}(s', a'(s'))
L(phi_1, D) = E[ (Q_phi_1(s,a) - y)^2 ]
L(phi_2, D) = E[ (Q_phi_2(s,a) - y)^2 ]
```
**Using the smaller Q-value for the target, and regressing toward that, helps fend off
overestimation** — a pessimistic estimate cannot be exploited the way an optimistic one can.
- **Trick Two: "Delayed" Policy Updates.** Update the policy (and the target networks) **less
frequently than the Q-function**. **The paper recommends one policy update for every two
Q-function updates.** This **damps the volatility that normally arises in DDPG because of
how a policy update changes the target.** Controlled by `policy_delay`.
- **Trick Three: Target Policy Smoothing.** Add **clipped** noise to each dimension of the
target action, then clip the result back into the valid action range:
```
a'(s') = clip( mu_theta_targ(s') + clip(epsilon, -c, c), a_Low, a_High ), epsilon ~ N(0, sigma)
```
**This is a regularizer.** It addresses a specific DDPG failure: **if the Q-function
approximator develops an incorrect sharp peak for some actions, the policy will quickly
exploit that peak and then have brittle or incorrect behavior.** Smoothing Q over similar
actions averts it.
- **Policy learning is unchanged from DDPG except in which critic it uses:** maximize
`E_{s~D}[ Q_phi_1(s, mu_theta(s)) ]` — **just the first Q-function.** (SAC differs here; see
ch19.)
## Reference Tables
| Trick | Fixes | Mechanism |
|-------|-------|-----------|
| Clipped double-Q | Q-value overestimation | Two critics, `min` of the two forms the shared target |
| Delayed policy updates | Volatility from policy updates moving the target | One policy + target update per `policy_delay` critic updates |
| Target policy smoothing | Policy exploiting sharp incorrect peaks in Q | Clipped Gaussian noise on the target action, then clip to action bounds |
| Hyperparameter | Role |
|----------------|------|
| `sigma` | Std dev of the target-smoothing noise |
| `c` | Clip bound on that noise |
| `policy_delay` | Q-updates per policy update; paper recommends 2 |
| `rho` (`polyak`) | Target network averaging coefficient, inherited from DDPG |
| `start_steps` | Initial uniform-random action steps, inherited from DDPG |
The loop is DDPG's, with two changes inside the update block: compute the smoothed target
action `a'(s')` first, update **both** Q-functions toward the clipped-double-Q target, and
then — **only when `j mod policy_delay == 0`** — take the policy ascent step and polyak-update
all three target networks.
## Exploration vs. Exploitation
Identical to DDPG: deterministic policy explored with **uncorrelated mean-zero Gaussian action
noise** at training time, optionally decayed (Spinning Up keeps it fixed), **no noise at test
time**, and a `start_steps` phase of uniform random actions at the beginning.
Note the distinction that is easy to blur: **exploration noise is added to the acting policy;
target policy smoothing noise is added to the target action inside the update.** Two different
noises, two different jobs.
## Mental Models
- **All three tricks are forms of pessimism or patience.** Take the smaller estimate, wait
longer before acting on the critic, and refuse to believe a Q-value that does not hold up
under a small action perturbation. Overestimation is the disease; conservatism is the cure.
- **Think of the policy as an adversary against your critic's errors.** Anything the critic
gets wrong in the optimistic direction is exactly what the policy will find. This is the
same failure shape as model bias in ch8's model-based methods.
- **"Together, these three tricks result in substantially improved performance over baseline
DDPG"** — they are presented as a package, and the exercise (ch12, 1.3) has you implement
the losses that contain all three.
## Anti-patterns
- **Implementing only clipped double-Q and calling it TD3.** The three tricks address three
distinct failure modes and the paper's result is for the package.
- **Using `min` over the two critics for the *policy* loss.** TD3 uses `Q_phi_1` alone for the
policy; using the min there is a SAC choice, not a TD3 one.
- **Updating the target networks on every critic step.** The target updates are inside the
`policy_delay` branch, along with the policy update.
## Key Takeaways
1. TD3 = DDPG + clipped double-Q + delayed policy updates + target policy smoothing.
2. The disease being treated is Q-value overestimation, which the policy actively exploits.
3. `min` of two critics forms the target; the policy maximizes `Q_phi_1` only.
4. Policy and target-network updates both live behind `policy_delay` (recommended 2).
5. Target smoothing noise is clipped twice: clip the noise to `[-c, c]`, then clip the
resulting action to the valid range.
6. Relevant paper: Fujimoto et al, 2018, *Addressing Function Approximation Error in
Actor-Critic Methods.*
## Connects To
- **Ch 17**: DDPG, whose brittleness is TD3's entire premise.
- **Ch 19**: SAC, which borrows clipped double-Q and gets smoothing for free from stochasticity.
- **Ch 12**: Exercise 1.3 asks you to write exactly these losses.
- **Ch 13**: TD3 is one of the three research-grade Spinning Up implementations.
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| reverse-engineer-anythingmorluto/reaReverse engineer native, managed, Electron/JavaScript, packaged, firmware, and browser targets with REA, and build agent environments (AgentEnv bundles) of web applications from a live instance or a capture, and of Linux desktop apps with a task from a request. Use shipped-artifact or requested runtime evidence to explain features, compare versions, decompile code, or guide a reconstruction. Skip REA for ordinary source-repository architecture analysis. | Stars: 472k | Updated | Claude CodeCodex | Verified | Install |
| diagnosing-superpowersobra/superpowersUse when a superpowers session went wrong and your human partner wants to know why — repeated work, ignored plans, stumbles, poor results, a skill that didn't fire, "it took too long", "why is it so expensive", "what is it doing" — or wants to build a bug report for the superpowers maintainers, for the current session or a past one identified by id or path, on any harness. | Stars: 323k | Updated | Claude CodeCodex | Verified | Install |
| dispatching-parallel-agentsobra/superpowersUse when facing 2+ independent tasks that can be worked on without shared state or sequential dependencies | Stars: 323k | Updated | Claude CodeCodex | Verified | Install |
| executing-plansobra/superpowersUse when executing an implementation plan in the current session as the implementer yourself — your human partner chose inline execution, or no subagent tool is available | Stars: 323k | Updated | Claude CodeCodex | Verified | Install |