Skip to content

alirezarezvani/claude-skills / engineering/deep-learning-book/skills/deep-learning-book

deep-learning-book

Verified

Use when studying or applying Goodfellow, Bengio and Courville's Deep Learning, or asking if its advice still holds. Indexes all 20 chapters with a 2016-to-2026 delta, plans reading paths with reading_path_planner.py and diagnoses training failures with training_diagnostics.py.

FileKindSize
SKILL.mdskill md10.8 KB
assets/chapter_worksheet.mdmarkdown2.3 KB
assets/example_layer_spec.jsonconfig1.2 KB
assets/study_log_template.mdmarkdown2.0 KB
chapters/ch01-introduction.mdmarkdown3.1 KB
chapters/ch02-linear-algebra.mdmarkdown3.1 KB
chapters/ch03-probability-information-theory.mdmarkdown3.6 KB
chapters/ch04-numerical-computation.mdmarkdown3.0 KB
chapters/ch05-machine-learning-basics.mdmarkdown3.5 KB
chapters/ch06-deep-feedforward-networks.mdmarkdown3.2 KB
chapters/ch07-regularization.mdmarkdown3.6 KB
chapters/ch08-optimization.mdmarkdown3.5 KB
chapters/ch09-convolutional-networks.mdmarkdown3.4 KB
chapters/ch10-sequence-modeling.mdmarkdown3.5 KB
chapters/ch11-practical-methodology.mdmarkdown3.2 KB
chapters/ch12-applications.mdmarkdown3.0 KB
chapters/ch13-linear-factor-models.mdmarkdown3.1 KB
chapters/ch14-autoencoders.mdmarkdown3.3 KB
chapters/ch15-representation-learning.mdmarkdown3.5 KB
chapters/ch16-structured-probabilistic-models.mdmarkdown3.2 KB
chapters/ch17-monte-carlo-methods.mdmarkdown3.0 KB
chapters/ch18-partition-function.mdmarkdown3.5 KB
chapters/ch19-approximate-inference.mdmarkdown3.3 KB
chapters/ch20-deep-generative-models.mdmarkdown3.7 KB
cheatsheet.mdmarkdown3.2 KB
glossary.mdmarkdown5.9 KB
patterns.mdmarkdown4.7 KB
references/book_to_2026_delta.mdmarkdown6.4 KB
references/prerequisite_map.mdmarkdown4.9 KB
references/rights_and_use.mdmarkdown4.4 KB
references/study_method_canon.mdmarkdown5.3 KB
scripts/capacity_planner.pyscript14.4 KB
scripts/model_arithmetic.pyscript16.9 KB
scripts/reading_path_planner.pyscript19.7 KB
scripts/training_diagnostics.pyscript14.1 KB

patterns.md

View on GitHub
# Patterns

Recurring techniques from the book, stated as instruments with their trade-offs. Each names the
chapter that develops it.

## Derive the loss from the output distribution (ch03, ch06)

Choose what p(y|x) is, then take its negative log. Gaussian → MSE, Bernoulli → binary
cross-entropy, categorical → cross-entropy, Laplace → MAE, mixture → mixture density loss.
**Trade-off:** none — this is strictly better than picking a loss by habit. **Failure mode it
prevents:** MSE on bounded, count-valued or heavily skewed targets.

## Work in log-space (ch04)

Sums of logs instead of products of probabilities; stabilized softmax and `log_softmax` rather
than hand-rolled exp/normalize. **Trade-off:** slightly more care at implementation time.
**Prevents:** silent underflow, NaN losses, and inf logits.

## Train-error-first triage (ch11, ch05)

Read training error before deciding anything. High training error → capacity or optimization is
the bottleneck; more data will not help. Low training error with high validation error → data or
regularization. **Trade-off:** requires honest, non-leaky splits. **Prevents:** the most expensive
common mistake, which is collecting data to fix underfitting.

## Overfit a tiny subset as a smoke test (ch11)

Take 10–50 examples and drive training loss to ~0. If you cannot, you have a bug — not a hard
problem. **Trade-off:** minutes. **Prevents:** weeks of tuning around a broken data pipeline,
a wrong loss reduction, or a detached gradient.

## Random search over grid search (ch11)

With more than about two hyperparameters, random search finds better configurations for the same
budget because it does not spend trials re-testing unimportant dimensions. **Trade-off:** results
are less tidy to tabulate. **Prevents:** exponential waste in dimensions that do not matter.

## Regularize in cost order (ch07)

More real data → augmentation with label-preserving transforms → early stopping → weight decay →
dropout → architecture change. Add one at a time and measure the train/val gap after each.
**Trade-off:** slower than stacking everything. **Prevents:** an unattributable result you cannot
tune.

## Parameter sharing over parameter penalties (ch07, ch09)

When a real invariance exists, encode it structurally (convolution, weight tying) rather than
penalizing a free parameter into behaving. **Trade-off:** the prior is hard — wrong invariance
means a wrong model, with no way for data to override it. **Prevents:** paying for capacity you
then have to regularize away.

## Diagnose by gradient behaviour (ch08, ch04)

Gradient norm exploding → clip. Norm large, loss flat → ill-conditioning; use momentum, an
adaptive optimizer, or normalization. Norm near zero with high loss → saturation or dead units;
check initialization and activations. Loss NaN → numerics before modelling. **Trade-off:**
requires instrumentation. **Prevents:** changing the architecture to fix an optimizer problem.

## Tune learning rate and schedule before optimizer family (ch08)

The schedule usually dominates the choice among SGD/Adam variants. **Trade-off:** none. **Prevents:**
optimizer-shopping while the real problem is a step size two orders of magnitude off.

## Name what prevents the identity map (ch14)

Every autoencoder needs an answer: bottleneck, sparsity penalty, input corruption, or Jacobian
contraction. **Trade-off:** each constraint defines a different notion of useful.
**Prevents:** an overcomplete autoencoder that learns a copy and reports a low loss.

## Escape the partition function deliberately (ch18, ch16)

Four routes: sample the negative phase (CD/PCD), sidestep Z algebraically (pseudolikelihood,
score matching), learn around it (NCE), or estimate it for evaluation only (AIS). **Trade-off:**
each optimizes a different objective, so they are not interchangeable. **Prevents:** discovering
mid-project that your undirected model cannot be trained or compared.

## Pick your generative failure mode (ch20, ch03)

Likelihood-based objectives cover modes and blur. Adversarial objectives sharpen and drop modes.
This follows from KL direction, so it is a design choice, not bad luck. **Trade-off:** you must
decide which error your application tolerates. **Prevents:** treating blurriness as a bug to be
tuned away.

## Evaluate representations by transfer, not reconstruction (ch15, ch14)

Linear probes and downstream task performance. **Trade-off:** needs a downstream task.
**Prevents:** optimizing reconstruction error into a code that memorized the input.

## Report Monte Carlo estimates with error bars (ch17)

Standard error and a mixing diagnostic, always. **Trade-off:** more reporting.
**Prevents:** a confidently unimodal answer from a chain that never left its starting mode.
slack-gif-creatoranthropics/skillsKnowledge and utilities for creating animated GIFs optimized for Slack. Provides constraints, validation tools, and animation concepts. Use when users request animated GIFs for Slack like "make me a GIF of X doing Y for Slack."Stars: 198kUpdated Claude CodeCodexVerifiedInstall
ponytailDietrichGebert/ponytailLazy senior dev mode for any coding task: the smallest change that fully solves it, and a reply a busy human understands in one read.Stars: 194kUpdated Claude CodeCodexVerifiedInstall
ponytail-debtDietrichGebert/ponytailHarvest every shortcut: comment into one debt ledger, so deferrals get tracked instead of forgotten. One-shot report.Stars: 194kUpdated Claude CodeCodexVerifiedInstall
ponytail-debtDietrichGebert/ponytailList every `shortcut:` comment (and older `ponytail:` ones) as a debt ledger. One-shot report, changes nothing. Use for "ponytail debt", "what did ponytail defer", "list the shortcuts", /ponytail-debt.Stars: 194kUpdated Claude CodeCodexVerifiedInstall