alirezarezvani/claude-skills / engineering/spinning-up-deep-rl/skills/spinning-up-deep-rl
spinning-up-deep-rl
VerifiedLearn or apply deep reinforcement learning with OpenAI Spinning Up: policy gradients, VPG, PPO, SAC, TD3, DDPG, TRPO. Use whenever an RL agent will not learn, reward stays flat, or the user asks how PPO or SAC works.
glossary.md
View on GitHub# Glossary
**Action space** — the set of all valid actions in an environment; discrete or continuous (Ch 7)
**Action-value function** — see Q-function (Ch 7)
**Advantage function** — `A^pi(s,a) = Q^pi(s,a) - V^pi(s)`; how much better a specific action is than randomly selecting one per pi (Ch 7, Ch 9)
**Agent** — the learner and decider; often used interchangeably with "policy" (Ch 7)
**Backtracking line search** — shrink the step by `alpha^j` until the exact KL constraint holds and surrogate advantage is positive (Ch 15)
**Baseline** — any state-only function `b(s)` subtractable from the policy gradient weight without changing its expectation (Ch 9)
**Bellman backup** — the right-hand side of a Bellman equation: reward plus next value (Ch 7)
**Bellman equation** — self-consistency condition: the value of your starting point is the reward you expect from being there plus the value of where you land next (Ch 7)
**Categorical policy** — stochastic policy for discrete actions, built like a classifier: logits then softmax (Ch 7)
**Clipped double-Q learning** — learn two Q-functions and use the smaller value to form the target (Ch 18, Ch 19)
**Deadly triad** — function approximation + bootstrapping + off-policy data, jointly causing instability in value learning (Ch 8)
**Diagonal Gaussian policy** — stochastic policy for continuous actions; mean network plus a diagonal covariance represented as a vector of log stds (Ch 7)
**Discount factor (gamma)** — in (0,1); makes an infinite reward sum converge and encodes "cash now beats cash later" (Ch 7)
**EGLP lemma** — Expected Grad-Log-Prob: `E_{x~P_theta}[grad log P_theta(x)] = 0` (Ch 9)
**Entropy** — `H(P) = E_{x~P}[-log P(x)]`; how random a random variable is (Ch 19)
**Entropy regularization** — adding an entropy bonus to the reward at each timestep, scaled by `alpha` (Ch 19)
**Epoch** — one experience-collection phase plus one policy update (Ch 9)
**ExperimentGrid** — Spinning Up's hyperparameter ablation tool; backs the CLI (Ch 4, Ch 20)
**Fully / partially observed** — whether the agent sees the complete state or only part of it (Ch 7)
**GAE** — Generalized Advantage Estimation; used by Spinning Up's VPG, TRPO and PPO (Ch 9)
**Log-derivative trick** — `grad P = P * grad log P` (Ch 9)
**Markov property** — transitions depend only on the most recent state and action (Ch 7)
**MDP** — Markov Decision Process, the 5-tuple `<S, A, R, P, rho_0>` (Ch 7)
**Model** — a function predicting state transitions and rewards (Ch 8)
**Model-based / model-free** — whether the agent has or learns a model (Ch 8)
**MPC** — model-predictive control: plan over a window, execute the first action, discard the rest, re-plan (Ch 8)
**MSBE** — Mean-Squared Bellman Error, the loss deep Q-learning minimizes (Ch 17)
**Natural Policy Gradient** — TRPO's analytic solution before the line search is added (Ch 15)
**Observation** — a partial description of a state, possibly omitting information (Ch 7)
**Off-policy** — updates may use data collected at any point in training (Ch 3, Ch 8)
**On-policy** — updates use only data from the most recent policy (Ch 3, Ch 8)
**Optimal policy (pi\*)** — `argmax_pi J(pi)` (Ch 7)
**Parameterized policy** — a policy whose output depends on adjustable parameters, e.g. network weights (Ch 7)
**Performance** — the plotter's family-aware alias, resolving to `AverageEpRet` (on-policy) or `AverageTestEpRet` (off-policy) (Ch 6)
**Policy** — the rule for choosing actions; deterministic `mu(s)` or stochastic `pi(.|s)` (Ch 7)
**Policy delay** — TD3's ratio of Q-updates to policy updates; recommended 2 (Ch 18)
**Policy gradient** — `grad_theta J(pi_theta)` (Ch 9)
**Polyak averaging** — target update `phi_targ <- rho*phi_targ + (1-rho)*phi`; `polyak` in the code (Ch 17)
**Q-function** — `Q^pi(s,a)`, expected return from taking a in s then following pi (Ch 7)
**Replay buffer** — the stored set of past transitions that off-policy algorithms sample from (Ch 17)
**Reparameterization trick** — sample by a deterministic function of state, parameters and independent noise (Ch 19)
**Return** — cumulative reward over a trajectory; finite-horizon undiscounted or infinite-horizon discounted (Ch 7)
**Reward-to-go** — `Rhat_t = sum_{t'=t}^T R(s_t', a_t', s_{t'+1})` (Ch 9)
**Silent failure** — broken RL code that runs correctly and simply never learns (Ch 10, Ch 12)
**Squashed Gaussian policy** — a Gaussian passed through `tanh` to bound actions; SAC's policy (Ch 19)
**start_steps** — initial phase of uniform-random actions used by DDPG, TD3 and SAC (Ch 17)
**State** — a complete description of the world, with nothing hidden (Ch 7)
**Surrogate advantage** — importance-weighted advantage measuring new-policy performance on old-policy data (Ch 15)
**Surrogate objective** — a local approximation of `J(pi_theta)` that is safe to maximize (Ch 8, Ch 16)
**Target network** — a time-lagged copy of a network used to compute stable Bellman targets (Ch 17)
**Target policy smoothing** — clipped noise added to the target action to prevent exploiting sharp Q peaks (Ch 18)
**Trajectory (tau)** — a sequence of states and actions; also called episode or rollout (Ch 7)
**Trust region** — a KL-divergence bound on how far the new policy may move from the old (Ch 15)
**Value function** — `V^pi(s)`, expected return from s following pi (Ch 7)
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| reverse-engineer-anythingmorluto/reaReverse engineer native, managed, Electron/JavaScript, packaged, firmware, and browser targets with REA, and build agent environments (AgentEnv bundles) of web applications from a live instance or a capture, and of Linux desktop apps with a task from a request. Use shipped-artifact or requested runtime evidence to explain features, compare versions, decompile code, or guide a reconstruction. Skip REA for ordinary source-repository architecture analysis. | Stars: 472k | Updated | Claude CodeCodex | Verified | Install |
| diagnosing-superpowersobra/superpowersUse when a superpowers session went wrong and your human partner wants to know why — repeated work, ignored plans, stumbles, poor results, a skill that didn't fire, "it took too long", "why is it so expensive", "what is it doing" — or wants to build a bug report for the superpowers maintainers, for the current session or a past one identified by id or path, on any harness. | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |
| dispatching-parallel-agentsobra/superpowersUse when facing 2+ independent tasks that can be worked on without shared state or sequential dependencies | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |
| executing-plansobra/superpowersUse when executing an implementation plan in the current session as the implementer yourself — your human partner chose inline execution, or no subagent tool is available | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |