Skip to content

alirezarezvani/claude-skills / engineering/deep-learning-book/skills/deep-learning-book

deep-learning-book

Verified

Use when studying or applying Goodfellow, Bengio and Courville's Deep Learning, or asking if its advice still holds. Indexes all 20 chapters with a 2016-to-2026 delta, plans reading paths with reading_path_planner.py and diagnoses training failures with training_diagnostics.py.

FileKindSize
SKILL.mdskill md10.8 KB
assets/chapter_worksheet.mdmarkdown2.3 KB
assets/example_layer_spec.jsonconfig1.2 KB
assets/study_log_template.mdmarkdown2.0 KB
chapters/ch01-introduction.mdmarkdown3.1 KB
chapters/ch02-linear-algebra.mdmarkdown3.1 KB
chapters/ch03-probability-information-theory.mdmarkdown3.6 KB
chapters/ch04-numerical-computation.mdmarkdown3.0 KB
chapters/ch05-machine-learning-basics.mdmarkdown3.5 KB
chapters/ch06-deep-feedforward-networks.mdmarkdown3.2 KB
chapters/ch07-regularization.mdmarkdown3.6 KB
chapters/ch08-optimization.mdmarkdown3.5 KB
chapters/ch09-convolutional-networks.mdmarkdown3.4 KB
chapters/ch10-sequence-modeling.mdmarkdown3.5 KB
chapters/ch11-practical-methodology.mdmarkdown3.2 KB
chapters/ch12-applications.mdmarkdown3.0 KB
chapters/ch13-linear-factor-models.mdmarkdown3.1 KB
chapters/ch14-autoencoders.mdmarkdown3.3 KB
chapters/ch15-representation-learning.mdmarkdown3.5 KB
chapters/ch16-structured-probabilistic-models.mdmarkdown3.2 KB
chapters/ch17-monte-carlo-methods.mdmarkdown3.0 KB
chapters/ch18-partition-function.mdmarkdown3.5 KB
chapters/ch19-approximate-inference.mdmarkdown3.3 KB
chapters/ch20-deep-generative-models.mdmarkdown3.7 KB
cheatsheet.mdmarkdown3.2 KB
glossary.mdmarkdown5.9 KB
patterns.mdmarkdown4.7 KB
references/book_to_2026_delta.mdmarkdown6.4 KB
references/prerequisite_map.mdmarkdown4.9 KB
references/rights_and_use.mdmarkdown4.4 KB
references/study_method_canon.mdmarkdown5.3 KB
scripts/capacity_planner.pyscript14.4 KB
scripts/model_arithmetic.pyscript16.9 KB
scripts/reading_path_planner.pyscript19.7 KB
scripts/training_diagnostics.pyscript14.1 KB

chapters/ch04-numerical-computation.md

View on GitHub
# Chapter 4: Numerical Computation

**Source chapter (free, official):** https://www.deeplearningbook.org/contents/numerical.html

## Core Idea

Real arithmetic on finite hardware fails in specific, predictable ways — underflow, overflow,
ill-conditioning — and gradient-based optimization inherits every one of them. This is the
shortest chapter with the highest debugging payoff.

## Key Concepts

- **Underflow / overflow**: numbers rounding to zero (then dividing) or exceeding representable
  range (then becoming inf/NaN).
- **Stabilized softmax**: subtract the max logit before exponentiating; mathematically identical,
  numerically survivable. `log_softmax` exists for the same reason.
- **Conditioning**: how much a function's output moves for small input moves; for a matrix, the
  condition number from Ch 2.
- **Gradient descent** and the first-order Taylor picture; **critical points**: minima, maxima,
  saddles.
- **Jacobian and Hessian**; second-order Taylor expansion, and the optimal step size implied by
  curvature.
- **Newton's method** and why it is not the default in deep learning: the Hessian is n×n in the
  parameter count.
- **Constrained optimization / KKT**: the framing that makes Ch 7's norm penalties readable as
  constraints.

## Mental Models

- Read a **NaN in the loss** as a numerics report first and a modelling bug second: check
  log(0), division by a near-zero denominator, exp of a large logit, and an exploding gradient
  in that order.
- Think of the Hessian's **eigenvalue spread as the terrain**: a large condition number is a
  narrow ravine, and the largest safe step size is set by the largest eigenvalue while progress
  is set by the smallest.
- Prefer the **log-domain** whenever probabilities are multiplied — this is the single highest
  yield habit in this chapter.

## Anti-patterns

- **Hand-rolling softmax or cross-entropy** in a training loop instead of using the fused,
  stabilized primitive.
- **Blaming the learning rate for every divergence**: ill-conditioning produces the same symptom
  and does not respond to the same fix.
- **Reaching for a second-order optimizer** in a model with millions of parameters without
  understanding the memory cost.

## What changed after 2016

Mixed-precision training (fp16 with loss scaling; then bf16, which trades mantissa bits for
exponent range specifically to avoid these failures) made this chapter's content a daily
operational concern rather than a background caution. Gradient clipping became standard practice
for transformer training. **Confidence: high.**

## Key Takeaways

1. Work in log-space by default for anything probabilistic.
2. When training diverges, separate "step too large" from "problem ill-conditioned" — they need
   different fixes.
3. Understand KKT well enough to read weight decay as a constraint; Ch 7 assumes it.

## Connects To

- **Ch 8**: every optimization difficulty named here recurs there at scale.
- **Ch 7**: the constrained-optimization view of regularization.
- **scripts/training_diagnostics.py**: mechanizes the NaN/divergence triage above.
slack-gif-creatoranthropics/skillsKnowledge and utilities for creating animated GIFs optimized for Slack. Provides constraints, validation tools, and animation concepts. Use when users request animated GIFs for Slack like "make me a GIF of X doing Y for Slack."Stars: 198kUpdated Claude CodeCodexVerifiedInstall
ponytailDietrichGebert/ponytailLazy senior dev mode for any coding task: the smallest change that fully solves it, and a reply a busy human understands in one read.Stars: 194kUpdated Claude CodeCodexVerifiedInstall
ponytail-debtDietrichGebert/ponytailHarvest every shortcut: comment into one debt ledger, so deferrals get tracked instead of forgotten. One-shot report.Stars: 194kUpdated Claude CodeCodexVerifiedInstall
ponytail-debtDietrichGebert/ponytailList every `shortcut:` comment (and older `ponytail:` ones) as a debt ledger. One-shot report, changes nothing. Use for "ponytail debt", "what did ponytail defer", "list the shortcuts", /ponytail-debt.Stars: 194kUpdated Claude CodeCodexVerifiedInstall