Skip to content

alirezarezvani/claude-skills / engineering/deep-learning-book/skills/deep-learning-book

deep-learning-book

Verified

Use when studying or applying Goodfellow, Bengio and Courville's Deep Learning, or asking if its advice still holds. Indexes all 20 chapters with a 2016-to-2026 delta, plans reading paths with reading_path_planner.py and diagnoses training failures with training_diagnostics.py.

FileKindSize
SKILL.mdskill md10.8 KB
assets/chapter_worksheet.mdmarkdown2.3 KB
assets/example_layer_spec.jsonconfig1.2 KB
assets/study_log_template.mdmarkdown2.0 KB
chapters/ch01-introduction.mdmarkdown3.1 KB
chapters/ch02-linear-algebra.mdmarkdown3.1 KB
chapters/ch03-probability-information-theory.mdmarkdown3.6 KB
chapters/ch04-numerical-computation.mdmarkdown3.0 KB
chapters/ch05-machine-learning-basics.mdmarkdown3.5 KB
chapters/ch06-deep-feedforward-networks.mdmarkdown3.2 KB
chapters/ch07-regularization.mdmarkdown3.6 KB
chapters/ch08-optimization.mdmarkdown3.5 KB
chapters/ch09-convolutional-networks.mdmarkdown3.4 KB
chapters/ch10-sequence-modeling.mdmarkdown3.5 KB
chapters/ch11-practical-methodology.mdmarkdown3.2 KB
chapters/ch12-applications.mdmarkdown3.0 KB
chapters/ch13-linear-factor-models.mdmarkdown3.1 KB
chapters/ch14-autoencoders.mdmarkdown3.3 KB
chapters/ch15-representation-learning.mdmarkdown3.5 KB
chapters/ch16-structured-probabilistic-models.mdmarkdown3.2 KB
chapters/ch17-monte-carlo-methods.mdmarkdown3.0 KB
chapters/ch18-partition-function.mdmarkdown3.5 KB
chapters/ch19-approximate-inference.mdmarkdown3.3 KB
chapters/ch20-deep-generative-models.mdmarkdown3.7 KB
cheatsheet.mdmarkdown3.2 KB
glossary.mdmarkdown5.9 KB
patterns.mdmarkdown4.7 KB
references/book_to_2026_delta.mdmarkdown6.4 KB
references/prerequisite_map.mdmarkdown4.9 KB
references/rights_and_use.mdmarkdown4.4 KB
references/study_method_canon.mdmarkdown5.3 KB
scripts/capacity_planner.pyscript14.4 KB
scripts/model_arithmetic.pyscript16.9 KB
scripts/reading_path_planner.pyscript19.7 KB
scripts/training_diagnostics.pyscript14.1 KB

chapters/ch08-optimization.md

View on GitHub
# Chapter 8: Optimization for Training Deep Models

**Source chapter (free, official):** https://www.deeplearningbook.org/contents/optimization.html

## Core Idea

Training is not pure optimization: you minimize an empirical surrogate to reduce a risk you
cannot measure, on a non-convex landscape, with noisy gradients. The chapter separates the
difficulties (ill-conditioning, saddles, cliffs, long-term dependencies, poor correspondence
between local and global structure) from the algorithms that address them.

## Frameworks Introduced

- **Empirical risk minimization and its surrogates**: you optimize a differentiable proxy, not
  the metric you report.
- **Minibatch SGD**: gradient noise scales roughly with 1/√batch, so bigger batches buy accuracy
  in the gradient at linear cost — a poor trade past a point.
- **Ill-conditioning**: the dominant obstacle in practice; gradient norm can *grow* while the loss
  stalls.
- **Saddle points, not local minima**: in high dimensions, critical points are overwhelmingly
  saddles. This reframes the folklore fear of local minima.
- **Cliffs and exploding gradients** → **gradient clipping**.
- **Momentum** and **Nesterov momentum**: accumulate a velocity to cross ravines.
- **Initialization**: Xavier/Glorot and He scaling to keep activation and gradient variance
  roughly constant with depth. Initialization is an algorithm, not a detail.
- **Adaptive methods**: AdaGrad (decaying), RMSProp, **Adam** (RMSProp + momentum + bias
  correction).
- **Second-order methods**: Newton, conjugate gradient, BFGS/L-BFGS — and why they rarely survive
  contact with minibatch noise and parameter counts.
- **Batch normalization**: reparameterization that stabilizes the scale of layer inputs.
- **Curriculum learning / coordinate descent / Polyak averaging**: the meta-strategies.

## Mental Models

- Diagnose by **what the gradient is doing**: norm exploding = cliff, clip. Norm large but loss
  flat = ill-conditioning, use momentum/adaptive/normalization. Norm ~0 with high loss = dead
  units or saturation.
- Treat **learning-rate schedule as a first-class hyperparameter** — usually more important than
  the choice among SGD/Adam variants.
- Read batch norm as **making the loss surface better conditioned**; whether it does so by
  reducing "internal covariate shift" is contested (Santurkar et al. 2018, post-2016).

## Anti-patterns

- **Fearing local minima**: spend the debugging effort on conditioning and learning rate.
- **Tuning the optimizer before the learning rate.**
- **Batch norm with tiny batches** — the batch statistics become noise. Use group/layer norm.

## What changed after 2016

The largest delta in Part II. **AdamW** replaced Adam+L2. Learning-rate **warmup plus cosine
decay** became standard for transformers. **LayerNorm** (and RMSNorm) displaced batch norm in
sequence models; pre-norm residual placement displaced post-norm for deep stacks. Large-batch
training with LARS/LAMB, and the linear-scaling-rule literature, arrived after the book.
**Confidence: high.**

## Key Takeaways

1. Tune learning rate and schedule first; optimizer family second.
2. Read gradient-norm behaviour before changing the architecture.
3. Use He/Xavier-style initialization deliberately — bad init is a common silent failure.

## Connects To

- **Ch 4**: conditioning, Hessians, and step size.
- **Ch 10**: exploding/vanishing gradients as the sequence-model version of these problems.
- **scripts/training_diagnostics.py**: this chapter's triage as an executable decision tree.
slack-gif-creatoranthropics/skillsKnowledge and utilities for creating animated GIFs optimized for Slack. Provides constraints, validation tools, and animation concepts. Use when users request animated GIFs for Slack like "make me a GIF of X doing Y for Slack."Stars: 198kUpdated Claude CodeCodexVerifiedInstall
ponytailDietrichGebert/ponytailLazy senior dev mode for any coding task: the smallest change that fully solves it, and a reply a busy human understands in one read.Stars: 194kUpdated Claude CodeCodexVerifiedInstall
ponytail-debtDietrichGebert/ponytailHarvest every shortcut: comment into one debt ledger, so deferrals get tracked instead of forgotten. One-shot report.Stars: 194kUpdated Claude CodeCodexVerifiedInstall
ponytail-debtDietrichGebert/ponytailList every `shortcut:` comment (and older `ponytail:` ones) as a debt ledger. One-shot report, changes nothing. Use for "ponytail debt", "what did ponytail defer", "list the shortcuts", /ponytail-debt.Stars: 194kUpdated Claude CodeCodexVerifiedInstall