Skip to content

alirezarezvani/claude-skills / engineering/deep-learning-book/skills/deep-learning-book

deep-learning-book

Verified

Use when studying or applying Goodfellow, Bengio and Courville's Deep Learning, or asking if its advice still holds. Indexes all 20 chapters with a 2016-to-2026 delta, plans reading paths with reading_path_planner.py and diagnoses training failures with training_diagnostics.py.

FileKindSize
SKILL.mdskill md10.8 KB
assets/chapter_worksheet.mdmarkdown2.3 KB
assets/example_layer_spec.jsonconfig1.2 KB
assets/study_log_template.mdmarkdown2.0 KB
chapters/ch01-introduction.mdmarkdown3.1 KB
chapters/ch02-linear-algebra.mdmarkdown3.1 KB
chapters/ch03-probability-information-theory.mdmarkdown3.6 KB
chapters/ch04-numerical-computation.mdmarkdown3.0 KB
chapters/ch05-machine-learning-basics.mdmarkdown3.5 KB
chapters/ch06-deep-feedforward-networks.mdmarkdown3.2 KB
chapters/ch07-regularization.mdmarkdown3.6 KB
chapters/ch08-optimization.mdmarkdown3.5 KB
chapters/ch09-convolutional-networks.mdmarkdown3.4 KB
chapters/ch10-sequence-modeling.mdmarkdown3.5 KB
chapters/ch11-practical-methodology.mdmarkdown3.2 KB
chapters/ch12-applications.mdmarkdown3.0 KB
chapters/ch13-linear-factor-models.mdmarkdown3.1 KB
chapters/ch14-autoencoders.mdmarkdown3.3 KB
chapters/ch15-representation-learning.mdmarkdown3.5 KB
chapters/ch16-structured-probabilistic-models.mdmarkdown3.2 KB
chapters/ch17-monte-carlo-methods.mdmarkdown3.0 KB
chapters/ch18-partition-function.mdmarkdown3.5 KB
chapters/ch19-approximate-inference.mdmarkdown3.3 KB
chapters/ch20-deep-generative-models.mdmarkdown3.7 KB
cheatsheet.mdmarkdown3.2 KB
glossary.mdmarkdown5.9 KB
patterns.mdmarkdown4.7 KB
references/book_to_2026_delta.mdmarkdown6.4 KB
references/prerequisite_map.mdmarkdown4.9 KB
references/rights_and_use.mdmarkdown4.4 KB
references/study_method_canon.mdmarkdown5.3 KB
scripts/capacity_planner.pyscript14.4 KB
scripts/model_arithmetic.pyscript16.9 KB
scripts/reading_path_planner.pyscript19.7 KB
scripts/training_diagnostics.pyscript14.1 KB

chapters/ch06-deep-feedforward-networks.md

View on GitHub
# Chapter 6: Deep Feedforward Networks

**Source chapter (free, official):** https://www.deeplearningbook.org/contents/mlp.html

## Core Idea

An MLP is a composed chain of parametric functions trained by gradient descent, where the choice
of output unit follows from the assumed output distribution and the choice of hidden unit follows
from gradient behaviour. Backpropagation is not learning; it is the chain rule scheduled well.

## Key Concepts

- **Universal approximation**: one sufficiently wide hidden layer can approximate any continuous
  function on a compact set. It says nothing about learnability, sample efficiency, or the size
  required — depth is how you get the same function with far fewer units.
- **Output units follow the distribution**: linear + MSE → Gaussian; sigmoid + binary
  cross-entropy → Bernoulli; softmax + cross-entropy → categorical; mixture density → multimodal.
- **Hidden units**: ReLU as the sane default; leaky/parametric ReLU, ELU, GELU/Swish (post-2016)
  for smoothness; sigmoid/tanh only where saturation is desired (gates).
- **Saturation**: sigmoid/tanh gradients vanish in the tails, which is why they are poor hidden
  units and fine as gates.
- **Backpropagation**: reverse-mode automatic differentiation over the computation graph. Cost is
  roughly one forward pass, memory is the stored activations.
- **Computation graph / autodiff**: the abstraction every framework implements. Forward mode is
  cheap in inputs; reverse mode is cheap in outputs — losses are scalar, hence reverse.

## Mental Models

- Pick the output unit from the **likelihood** (Ch 3), then pick the loss as its negative log.
  Every "which loss?" question reduces to "which distribution?"
- Read **ReLU's advantage as gradient preservation**, not nonlinearity per se: it is piecewise
  linear, so the gradient through an active unit is exactly 1.
- Treat activation memory as the **real** cost of depth in training: the backward pass needs the
  forward activations, which is why checkpointing trades compute for memory.

## Anti-patterns

- **Citing universal approximation to justify a shallow model** — the theorem allows an
  exponentially wide layer, which is not an engineering plan.
- **Sigmoid hidden layers** in a deep stack.
- **Pairing a sigmoid output with MSE**: gradients vanish exactly where the model is most wrong.
  Use cross-entropy.

## What changed after 2016

Smooth activations (GELU, SiLU/Swish) became the default in transformers; gated variants
(GLU, SwiGLU) are now standard in large language model feedforward blocks. Residual connections
(He et al. 2015) are mentioned here only in passing but became the structural default for every
deep stack. Reverse-mode autodiff is unchanged. **Confidence: high.**

## Key Takeaways

1. Derive the loss from the output distribution rather than picking it by habit.
2. Default to ReLU-family hidden units; reserve saturating units for gates.
3. Budget activation memory as a first-class constraint, not an implementation detail.

## Connects To

- **Ch 3**: the distributions that determine output units.
- **Ch 8**: why gradient preservation matters at depth.
- **scripts/model_arithmetic.py**: parameter, FLOP and activation-memory accounting for a stack.
slack-gif-creatoranthropics/skillsKnowledge and utilities for creating animated GIFs optimized for Slack. Provides constraints, validation tools, and animation concepts. Use when users request animated GIFs for Slack like "make me a GIF of X doing Y for Slack."Stars: 198kUpdated Claude CodeCodexVerifiedInstall
ponytailDietrichGebert/ponytailLazy senior dev mode for any coding task: the smallest change that fully solves it, and a reply a busy human understands in one read.Stars: 194kUpdated Claude CodeCodexVerifiedInstall
ponytail-debtDietrichGebert/ponytailHarvest every shortcut: comment into one debt ledger, so deferrals get tracked instead of forgotten. One-shot report.Stars: 194kUpdated Claude CodeCodexVerifiedInstall
ponytail-debtDietrichGebert/ponytailList every `shortcut:` comment (and older `ponytail:` ones) as a debt ledger. One-shot report, changes nothing. Use for "ponytail debt", "what did ponytail defer", "list the shortcuts", /ponytail-debt.Stars: 194kUpdated Claude CodeCodexVerifiedInstall