alirezarezvani/claude-skills / engineering/deep-learning-book/skills/deep-learning-book
deep-learning-book
VerifiedUse when studying or applying Goodfellow, Bengio and Courville's Deep Learning, or asking if its advice still holds. Indexes all 20 chapters with a 2016-to-2026 delta, plans reading paths with reading_path_planner.py and diagnoses training failures with training_diagnostics.py.
references/book_to_2026_delta.md
View on GitHub# The 2016 β 2026 Delta
The book was published in 2016. *Attention Is All You Need* appeared in 2017. Everything below
tracks what a reader should hold, discount, or replace β per chapter, with a confidence level on
each claim.
Confidence levels: π’ well-established and replicated Β· π‘ supported but contested or
context-dependent Β· π΄ folklore, named as folklore.
---
## Chapters that aged well (read as current)
| Ch | Why it holds |
|---|---|
| 2, 3, 4 | Mathematics does not age. π’ |
| 5 | The learning-theory frame is current β with one correction (double descent, below). π’ |
| 11 | The practical workflow became the backbone of MLOps practice. π’ |
| 16 | The directed/undirected taxonomy classifies models invented since, including transformers as chain-rule factorizations. π’ |
| 18 | Score matching became diffusion; NCE became contrastive learning. The chapter's ideas travelled further than its models. π’ |
## The five corrections that matter
### 1. Double descent qualifies the U-curve (ch05)
The classical capacity curve β error falls, then rises past the optimum β is incomplete. Past the
interpolation threshold, test error can descend a second time. Belkin et al. (2019) named it;
Nakkiran et al. (2020) showed model-wise, epoch-wise and sample-wise variants in deep networks.
The biasβvariance algebra is unchanged; the *advice* "shrink the model when it overfits" is no
longer the only correct move. π’
### 2. AdamW: weight decay β L2 under adaptive optimizers (ch07, ch08)
Adding an L2 term to the loss and applying weight decay to the update are equivalent for plain
SGD and **not** equivalent for Adam, because the adaptive denominator rescales the penalty.
Loshchilov & Hutter (2017/2019) decoupled them; AdamW is now the default. The book's Ch 7 treats
the two as interchangeable. π’
### 3. Transformers displaced recurrence (ch10, ch12)
Vaswani et al. (2017) removed recurrence entirely: attention gives an O(1) path between any two
positions and parallelizes over sequence length. What survives from Ch 10: the vanishing/exploding
gradient analysis, gradient clipping, teacher forcing, exposure bias. What is superseded: the
architecture recommendation. Note the return of linear-time recurrence in state-space models
(S4, 2021; Mamba, 2023) for long context β which makes Ch 10's analysis live again. π’
### 4. Diffusion displaced the Part III generative models (ch14, ch18, ch20)
The line runs directly through the book: denoising autoencoders (Ch 14) β denoising score
matching (Ch 18) β score-based generative models (Song & Ermon, 2019) β DDPM (Ho et al., 2020).
Diffusion now dominates image, audio and video generation; autoregressive transformers dominate
text. VAEs survive mainly as latent-space compressors inside latent-diffusion pipelines. GANs
remain useful for few-step generation. Boltzmann machines are historical. π’
### 5. Self-supervised learning vindicated Ch 15 while replacing its methods
The chapter's bet β that unsupervised representation learning would matter β was right, and every
specific method it lists was replaced. Contrastive (SimCLR, MoCo, CLIP), masked prediction (BERT,
MAE) and next-token prediction at scale are the modern routes. Two corrections: greedy layer-wise
pretraining is now purely historical π’, and unsupervised disentanglement was shown impossible
without inductive biases or supervision (Locatello et al., 2019) π’.
## Additions the book has no chapter for
| Topic | Status |
|---|---|
| **Neural scaling laws** (Kaplan 2020; Hoffmann 2022) | Quantifies Ch 1's "scale matters" and Ch 11's data-vs-capacity decision. π’ |
| **Normalization placement** β pre-norm vs post-norm residual, LayerNorm/RMSNorm over BatchNorm | Standard for deep sequence stacks. π’ |
| **Warmup + cosine decay schedules** | Standard for transformer training. π’ |
| **Calibration** (Guo et al., 2017) | Modern networks are systematically overconfident; temperature scaling is the cheap fix. Ch 3 does not mention it. π’ |
| **RLHF / DPO alignment** | Entirely outside the book. π’ |
| **Mixture-of-Experts** | The mature form of Ch 12's conditional computation. π’ |
| **Mixed precision (fp16 loss scaling, bf16)** | Made Ch 4's numerics an operational daily concern. π’ |
| **Lottery-ticket / pruning theory** (Frankle & Carbin, 2019) | Active; the strong form remains contested. π‘ |
| **"Batch norm works by reducing internal covariate shift"** | The original explanation; challenged by Santurkar et al. (2018), who attribute the effect to smoothing the loss landscape. Treat the mechanism as unsettled. π‘ |
| **"Local minima are the problem in deep nets"** | π΄ Folklore the book itself corrects: high-dimensional critical points are overwhelmingly saddles (Ch 8). |
| **"You need a GPU cluster to learn deep learning"** | π΄ Folklore. Every mechanism in Parts IβII is observable on a laptop-scale model. |
## How to use this file
When a chapter's advice conflicts with current practice, the conflict is almost always in the
**recommendation**, not the **analysis**. The book explains why things fail; the field has changed
what it reaches for. Keep the diagnosis, replace the prescription.
## Sources
1. Vaswani et al., "Attention Is All You Need," NeurIPS 2017 β arXiv:1706.03762.
2. Loshchilov & Hutter, "Decoupled Weight Decay Regularization," ICLR 2019 β arXiv:1711.05101.
3. Belkin, Hsu, Ma & Mandal, "Reconciling modern machine-learning practice and the classical
biasβvariance trade-off," PNAS 116(32), 2019; Nakkiran et al., "Deep Double Descent," ICLR 2020.
4. Ho, Jain & Abbeel, "Denoising Diffusion Probabilistic Models," NeurIPS 2020 β arXiv:2006.11239;
Song & Ermon, "Generative Modeling by Estimating Gradients of the Data Distribution,"
NeurIPS 2019.
5. Kaplan et al., "Scaling Laws for Neural Language Models," 2020 β arXiv:2001.08361;
Hoffmann et al., "Training Compute-Optimal Large Language Models," 2022 β arXiv:2203.15556.
6. Locatello et al., "Challenging Common Assumptions in the Unsupervised Learning of Disentangled
Representations," ICML 2019 (best paper) β arXiv:1811.12359.
7. Guo, Pleiss, Sun & Weinberger, "On Calibration of Modern Neural Networks," ICML 2017;
Santurkar et al., "How Does Batch Normalization Help Optimization?," NeurIPS 2018.
8. Dosovitskiy et al., "An Image is Worth 16x16 Words" (ViT), ICLR 2021 β arXiv:2010.11929;
Liu et al., "A ConvNet for the 2020s" (ConvNeXt), CVPR 2022.
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| slack-gif-creatoranthropics/skillsKnowledge and utilities for creating animated GIFs optimized for Slack. Provides constraints, validation tools, and animation concepts. Use when users request animated GIFs for Slack like "make me a GIF of X doing Y for Slack." | Stars: 198k | Updated | Claude CodeCodex | Verified | Install |
| ponytailDietrichGebert/ponytailLazy senior dev mode for any coding task: the smallest change that fully solves it, and a reply a busy human understands in one read. | Stars: 194k | Updated | Claude CodeCodex | Verified | Install |
| ponytail-debtDietrichGebert/ponytailHarvest every shortcut: comment into one debt ledger, so deferrals get tracked instead of forgotten. One-shot report. | Stars: 194k | Updated | Claude CodeCodex | Verified | Install |
| ponytail-debtDietrichGebert/ponytailList every `shortcut:` comment (and older `ponytail:` ones) as a debt ledger. One-shot report, changes nothing. Use for "ponytail debt", "what did ponytail defer", "list the shortcuts", /ponytail-debt. | Stars: 194k | Updated | Claude CodeCodex | Verified | Install |