Skip to content

alirezarezvani/claude-skills / engineering/deep-learning-book/skills/deep-learning-book

deep-learning-book

Verified

Use when studying or applying Goodfellow, Bengio and Courville's Deep Learning, or asking if its advice still holds. Indexes all 20 chapters with a 2016-to-2026 delta, plans reading paths with reading_path_planner.py and diagnoses training failures with training_diagnostics.py.

FileKindSize
SKILL.mdskill md10.8 KB
assets/chapter_worksheet.mdmarkdown2.3 KB
assets/example_layer_spec.jsonconfig1.2 KB
assets/study_log_template.mdmarkdown2.0 KB
chapters/ch01-introduction.mdmarkdown3.1 KB
chapters/ch02-linear-algebra.mdmarkdown3.1 KB
chapters/ch03-probability-information-theory.mdmarkdown3.6 KB
chapters/ch04-numerical-computation.mdmarkdown3.0 KB
chapters/ch05-machine-learning-basics.mdmarkdown3.5 KB
chapters/ch06-deep-feedforward-networks.mdmarkdown3.2 KB
chapters/ch07-regularization.mdmarkdown3.6 KB
chapters/ch08-optimization.mdmarkdown3.5 KB
chapters/ch09-convolutional-networks.mdmarkdown3.4 KB
chapters/ch10-sequence-modeling.mdmarkdown3.5 KB
chapters/ch11-practical-methodology.mdmarkdown3.2 KB
chapters/ch12-applications.mdmarkdown3.0 KB
chapters/ch13-linear-factor-models.mdmarkdown3.1 KB
chapters/ch14-autoencoders.mdmarkdown3.3 KB
chapters/ch15-representation-learning.mdmarkdown3.5 KB
chapters/ch16-structured-probabilistic-models.mdmarkdown3.2 KB
chapters/ch17-monte-carlo-methods.mdmarkdown3.0 KB
chapters/ch18-partition-function.mdmarkdown3.5 KB
chapters/ch19-approximate-inference.mdmarkdown3.3 KB
chapters/ch20-deep-generative-models.mdmarkdown3.7 KB
cheatsheet.mdmarkdown3.2 KB
glossary.mdmarkdown5.9 KB
patterns.mdmarkdown4.7 KB
references/book_to_2026_delta.mdmarkdown6.4 KB
references/prerequisite_map.mdmarkdown4.9 KB
references/rights_and_use.mdmarkdown4.4 KB
references/study_method_canon.mdmarkdown5.3 KB
scripts/capacity_planner.pyscript14.4 KB
scripts/model_arithmetic.pyscript16.9 KB
scripts/reading_path_planner.pyscript19.7 KB
scripts/training_diagnostics.pyscript14.1 KB

chapters/ch07-regularization.md

View on GitHub
# Chapter 7: Regularization for Deep Learning

**Source chapter (free, official):** https://www.deeplearningbook.org/contents/regularization.html

## Core Idea

Regularization is anything that trades training error for generalization error, and this chapter
is the catalogue: norm penalties, data augmentation, noise, early stopping, parameter sharing,
sparsity, ensembling, dropout, and adversarial training — with their equivalences made explicit.

## Frameworks Introduced

- **L2 / weight decay**: shrinks parameters along low-curvature Hessian directions. Under a
  quadratic approximation, it damps each eigen-direction by λᵢ/(λᵢ+α) — directions the loss does
  not care about get pulled to zero.
- **L1**: yields genuine sparsity; equivalent to a Laplace prior on parameters.
- **Norm penalty as constraint**: via KKT (Ch 4), a penalty is a soft version of a norm ball.
  Explicit projection (max-norm) is sometimes better behaved.
- **Dataset augmentation**: the most reliable regularizer when the invariance is real. Choose
  transformations that preserve the label — and check that they do.
- **Noise injection**: on inputs (≈ L2 penalty for some models), on weights (encourages flat
  minima), on labels (label smoothing).
- **Early stopping**: the cheapest regularizer; under a quadratic approximation it is
  approximately equivalent to L2 with a strength set by the number of steps.
- **Parameter tying / sharing**: the strongest form — it removes parameters instead of penalizing
  them. Convolution is parameter sharing (Ch 9).
- **Bagging and ensembles**: variance reduction by averaging independently trained models.
- **Dropout**: approximate ensembling over exponentially many sub-networks at the cost of one.
- **Adversarial training**: penalizing sensitivity to worst-case local perturbation.

## Mental Models

- Order regularizers by **cost per unit of gap closed**: more real data > augmentation > early
  stopping > weight decay > dropout > architecture surgery.
- Read weight decay as **"forget the directions the data does not constrain"** — that is exactly
  what the eigen-analysis says.
- Treat dropout as an **ensemble**, so its interaction with batch normalization (train/test
  statistics mismatch) is expected rather than surprising.

## Anti-patterns

- **Stacking every regularizer at once** and then tuning — you cannot attribute the effect.
- **Augmenting with label-destroying transforms** (horizontal flip on digits, aggressive crops on
  fine-grained classes).
- **Dropout inside a residual transformer block plus batch norm plus heavy weight decay** without
  measuring; modern stacks regularize far more lightly than 2016 practice.

## What changed after 2016

Weight decay and L2 were shown to be *not* equivalent under adaptive optimizers, which is why
**AdamW** (Loshchilov & Hutter 2017/2019) decouples them — this post-dates the book and is now
the default. Dropout largely left large-scale vision and language models, displaced by
normalization, augmentation and sheer data volume. Label smoothing and stochastic depth became
common. **Confidence: high** for AdamW; **high** for the decline of dropout at scale.

## Key Takeaways

1. Add regularizers one at a time and measure the gap after each.
2. Use decoupled weight decay (AdamW) rather than L2-in-the-loss with an adaptive optimizer.
3. Prefer parameter sharing over penalties when a real invariance exists.

## Connects To

- **Ch 5**: the bias–variance budget these all spend from.
- **Ch 8**: optimizer choice changes what weight decay means.
- **Ch 11**: which regularizer to reach for, given a measured gap.
slack-gif-creatoranthropics/skillsKnowledge and utilities for creating animated GIFs optimized for Slack. Provides constraints, validation tools, and animation concepts. Use when users request animated GIFs for Slack like "make me a GIF of X doing Y for Slack."Stars: 198kUpdated Claude CodeCodexVerifiedInstall
ponytailDietrichGebert/ponytailLazy senior dev mode for any coding task: the smallest change that fully solves it, and a reply a busy human understands in one read.Stars: 194kUpdated Claude CodeCodexVerifiedInstall
ponytail-debtDietrichGebert/ponytailHarvest every shortcut: comment into one debt ledger, so deferrals get tracked instead of forgotten. One-shot report.Stars: 194kUpdated Claude CodeCodexVerifiedInstall
ponytail-debtDietrichGebert/ponytailList every `shortcut:` comment (and older `ponytail:` ones) as a debt ledger. One-shot report, changes nothing. Use for "ponytail debt", "what did ponytail defer", "list the shortcuts", /ponytail-debt.Stars: 194kUpdated Claude CodeCodexVerifiedInstall