Skip to content

alirezarezvani/claude-skills / engineering/deep-learning-book/skills/deep-learning-book

deep-learning-book

Verified

Use when studying or applying Goodfellow, Bengio and Courville's Deep Learning, or asking if its advice still holds. Indexes all 20 chapters with a 2016-to-2026 delta, plans reading paths with reading_path_planner.py and diagnoses training failures with training_diagnostics.py.

FileKindSize
SKILL.mdskill md10.8 KB
assets/chapter_worksheet.mdmarkdown2.3 KB
assets/example_layer_spec.jsonconfig1.2 KB
assets/study_log_template.mdmarkdown2.0 KB
chapters/ch01-introduction.mdmarkdown3.1 KB
chapters/ch02-linear-algebra.mdmarkdown3.1 KB
chapters/ch03-probability-information-theory.mdmarkdown3.6 KB
chapters/ch04-numerical-computation.mdmarkdown3.0 KB
chapters/ch05-machine-learning-basics.mdmarkdown3.5 KB
chapters/ch06-deep-feedforward-networks.mdmarkdown3.2 KB
chapters/ch07-regularization.mdmarkdown3.6 KB
chapters/ch08-optimization.mdmarkdown3.5 KB
chapters/ch09-convolutional-networks.mdmarkdown3.4 KB
chapters/ch10-sequence-modeling.mdmarkdown3.5 KB
chapters/ch11-practical-methodology.mdmarkdown3.2 KB
chapters/ch12-applications.mdmarkdown3.0 KB
chapters/ch13-linear-factor-models.mdmarkdown3.1 KB
chapters/ch14-autoencoders.mdmarkdown3.3 KB
chapters/ch15-representation-learning.mdmarkdown3.5 KB
chapters/ch16-structured-probabilistic-models.mdmarkdown3.2 KB
chapters/ch17-monte-carlo-methods.mdmarkdown3.0 KB
chapters/ch18-partition-function.mdmarkdown3.5 KB
chapters/ch19-approximate-inference.mdmarkdown3.3 KB
chapters/ch20-deep-generative-models.mdmarkdown3.7 KB
cheatsheet.mdmarkdown3.2 KB
glossary.mdmarkdown5.9 KB
patterns.mdmarkdown4.7 KB
references/book_to_2026_delta.mdmarkdown6.4 KB
references/prerequisite_map.mdmarkdown4.9 KB
references/rights_and_use.mdmarkdown4.4 KB
references/study_method_canon.mdmarkdown5.3 KB
scripts/capacity_planner.pyscript14.4 KB
scripts/model_arithmetic.pyscript16.9 KB
scripts/reading_path_planner.pyscript19.7 KB
scripts/training_diagnostics.pyscript14.1 KB

cheatsheet.md

View on GitHub
# Cheatsheet

Decision rules and thresholds. One line each; the chapter has the reasoning.

## Loss selection (ch03, ch06)

| Target | Distribution | Output unit | Loss |
|---|---|---|---|
| Real, unbounded | Gaussian | linear | MSE |
| Binary | Bernoulli | sigmoid | binary cross-entropy |
| One of K | Categorical | softmax | cross-entropy |
| Real, heavy-tailed | Laplace | linear | MAE |
| Multimodal real | Mixture | mixture density | NLL of the mixture |

Never pair sigmoid output with MSE — the gradient vanishes where the model is most wrong.

## Underfit vs overfit (ch05, ch11)

| Train error | Val error | Verdict | First action |
|---|---|---|---|
| High | High | Underfit / optimization | Check LR, init, capacity — **not** more data |
| Low | High | Overfit | More data → augmentation → regularization |
| Low | Low | Working | Stop; check the metric is the right one |
| ~0 on 20 examples: fails | — | Bug | Fix the pipeline before anything else |

Past the interpolation threshold, prefer more data / longer training / more regularization over
shrinking the model (double descent, ch05).

## Optimizer defaults (ch08)

| Situation | Reach for |
|---|---|
| Any starting point | AdamW, LR ~1e-3 (small nets) / ~1e-4 (large), warmup + cosine decay |
| Recurrence in the graph | Add gradient clipping, always |
| Tiny batches | Group/layer norm, not batch norm |
| Loss flat, grad norm large | Ill-conditioning — momentum, adaptive, or normalization |
| Loss NaN | Numerics first: log(0), div by ~0, exp overflow, exploding grad |

## Regularization ladder (ch07)

More real data → label-preserving augmentation → early stopping → decoupled weight decay →
label smoothing → dropout → smaller model. One at a time; measure the gap after each.

## Architecture prior (ch09, ch10)

| Data | Prior that fits | Note |
|---|---|---|
| Grid, translation-invariant statistics | Convolution | Cheapest when data is limited |
| Sequence, long-range dependence | Attention/transformer | The 2016 RNN advice is superseded |
| Sequence, very long context | State-space / linear recurrence | Post-2016; ch10's gradient analysis applies |
| Arbitrary tabular columns | Neither | Locality prior is false |

## Generative model taxonomy (ch20)

| Family | Handles Z by | Characteristic failure |
|---|---|---|
| Autoregressive | Chain rule — no Z | Slow sequential sampling |
| VAE | Bounding likelihood (ELBO) | Blurry samples |
| GAN | Avoiding likelihood | Mode collapse, unstable training |
| Diffusion / score-based | Learning ∇ log p | Many sampling steps (mitigable) |
| Boltzmann machines | Sampling the negative phase | Historical; do not start here |

## Numerical hygiene (ch04)

Log-space for probability products · stabilized/fused softmax and cross-entropy · clip gradients
with recurrence · bf16 over fp16 when range matters · check condition number before blaming LR.

## Study-order rule (ch01–ch20)

Ch 5 and Ch 11 are the highest-value chapters for a practitioner and can be read early.
Ch 2–4 are prerequisites for Ch 8 specifically. Ch 13→14→19→20 is the only strict chain in
Part III. Ch 18 is worth reading even if you never train an undirected model — score matching
became diffusion.
slack-gif-creatoranthropics/skillsKnowledge and utilities for creating animated GIFs optimized for Slack. Provides constraints, validation tools, and animation concepts. Use when users request animated GIFs for Slack like "make me a GIF of X doing Y for Slack."Stars: 198kUpdated Claude CodeCodexVerifiedInstall
ponytailDietrichGebert/ponytailLazy senior dev mode for any coding task: the smallest change that fully solves it, and a reply a busy human understands in one read.Stars: 194kUpdated Claude CodeCodexVerifiedInstall
ponytail-debtDietrichGebert/ponytailHarvest every shortcut: comment into one debt ledger, so deferrals get tracked instead of forgotten. One-shot report.Stars: 194kUpdated Claude CodeCodexVerifiedInstall
ponytail-debtDietrichGebert/ponytailList every `shortcut:` comment (and older `ponytail:` ones) as a debt ledger. One-shot report, changes nothing. Use for "ponytail debt", "what did ponytail defer", "list the shortcuts", /ponytail-debt.Stars: 194kUpdated Claude CodeCodexVerifiedInstall