Skip to content

alirezarezvani/claude-skills / engineering/deep-learning-book/skills/deep-learning-book

deep-learning-book

Verified

Use when studying or applying Goodfellow, Bengio and Courville's Deep Learning, or asking if its advice still holds. Indexes all 20 chapters with a 2016-to-2026 delta, plans reading paths with reading_path_planner.py and diagnoses training failures with training_diagnostics.py.

FileKindSize
SKILL.mdskill md10.8 KB
assets/chapter_worksheet.mdmarkdown2.3 KB
assets/example_layer_spec.jsonconfig1.2 KB
assets/study_log_template.mdmarkdown2.0 KB
chapters/ch01-introduction.mdmarkdown3.1 KB
chapters/ch02-linear-algebra.mdmarkdown3.1 KB
chapters/ch03-probability-information-theory.mdmarkdown3.6 KB
chapters/ch04-numerical-computation.mdmarkdown3.0 KB
chapters/ch05-machine-learning-basics.mdmarkdown3.5 KB
chapters/ch06-deep-feedforward-networks.mdmarkdown3.2 KB
chapters/ch07-regularization.mdmarkdown3.6 KB
chapters/ch08-optimization.mdmarkdown3.5 KB
chapters/ch09-convolutional-networks.mdmarkdown3.4 KB
chapters/ch10-sequence-modeling.mdmarkdown3.5 KB
chapters/ch11-practical-methodology.mdmarkdown3.2 KB
chapters/ch12-applications.mdmarkdown3.0 KB
chapters/ch13-linear-factor-models.mdmarkdown3.1 KB
chapters/ch14-autoencoders.mdmarkdown3.3 KB
chapters/ch15-representation-learning.mdmarkdown3.5 KB
chapters/ch16-structured-probabilistic-models.mdmarkdown3.2 KB
chapters/ch17-monte-carlo-methods.mdmarkdown3.0 KB
chapters/ch18-partition-function.mdmarkdown3.5 KB
chapters/ch19-approximate-inference.mdmarkdown3.3 KB
chapters/ch20-deep-generative-models.mdmarkdown3.7 KB
cheatsheet.mdmarkdown3.2 KB
glossary.mdmarkdown5.9 KB
patterns.mdmarkdown4.7 KB
references/book_to_2026_delta.mdmarkdown6.4 KB
references/prerequisite_map.mdmarkdown4.9 KB
references/rights_and_use.mdmarkdown4.4 KB
references/study_method_canon.mdmarkdown5.3 KB
scripts/capacity_planner.pyscript14.4 KB
scripts/model_arithmetic.pyscript16.9 KB
scripts/reading_path_planner.pyscript19.7 KB
scripts/training_diagnostics.pyscript14.1 KB

chapters/ch03-probability-information-theory.md

View on GitHub
# Chapter 3: Probability and Information Theory

**Source chapter (free, official):** https://www.deeplearningbook.org/contents/prob.html

## Core Idea

Deep learning models are probability distributions, and losses are log-likelihoods. This chapter
installs the vocabulary — random variables, the common distributions, expectation, Bayes' rule,
entropy, KL divergence, cross-entropy — that makes "why this loss?" answerable rather than
conventional.

## Key Concepts

- **Frequentist vs Bayesian probability**: rates of events vs degrees of belief. Deep learning
  uses the Bayesian reading for parameters and the frequentist one for evaluation, mostly
  without saying so.
- **Marginal / conditional / chain rule**: the algebra every graphical model in Ch 16 runs on.
- **Common distributions**: Bernoulli, categorical, Gaussian, exponential, Laplace, Dirac,
  empirical, and mixtures. Each corresponds to an output layer you will actually build.
- **Self-information and Shannon entropy**: surprise, and expected surprise.
- **KL divergence**: asymmetric. D(p‖q) ≠ D(q‖p), and the asymmetry decides whether your fitted
  model covers all modes or concentrates on one.
- **Cross-entropy**: H(p,q) = H(p) + D(p‖q). Minimizing cross-entropy over q is minimizing KL,
  because H(p) is constant in q — this is why classification uses it.
- **Structured probabilistic models**: factorization of a joint into conditionals over a graph.

## Mental Models

- Read every loss as a **negative log-likelihood** under an assumed output distribution: MSE is
  a Gaussian with fixed variance, cross-entropy is a categorical, MAE is a Laplace. If you know
  the assumed distribution, you know when the loss is wrong for your data.
- Use KL **direction** as a design lever: forward KL (data ‖ model) is mode-covering and gives
  blurry averages; reverse KL (model ‖ data) is mode-seeking and gives sharp but partial fits.
  This one fact explains most of Ch 19 and 20.
- Treat softmax as **exp-then-normalize on logits**, and remember from Ch 4 that it must be
  computed in a shift-stabilized form.

## Anti-patterns

- **Choosing MSE for a bounded or count-valued target** — you have assumed a Gaussian on data
  that is not Gaussian, and the residual structure will tell you so.
- **Reading KL as a distance**: it is not symmetric and does not satisfy the triangle inequality.
- **Interpreting softmax outputs as calibrated probabilities** without checking calibration —
  the chapter's math does not promise calibration, and modern networks are typically
  overconfident (Guo et al. 2017, post-dating the book).

## What changed after 2016

The probabilistic core is unchanged. Two additions matter: the calibration literature (deep
networks are systematically overconfident; temperature scaling is the cheap fix), and the rise
of losses that are *not* clean log-likelihoods — contrastive/InfoNCE objectives, and preference
losses such as DPO. The chapter's "every loss is a likelihood" framing needs that caveat now.
**Confidence: high** for calibration; **high** for the contrastive family being outside the
chapter's scope.

## Key Takeaways

1. State the output distribution before choosing the loss; the loss follows from it.
2. When a generative model looks blurry, suspect forward KL; when it looks mode-collapsed,
   suspect reverse KL.
3. Check calibration separately from accuracy — the book does not, and the gap is real.

## Connects To

- **Ch 5**: maximum likelihood as the estimator that justifies these losses.
- **Ch 16–19**: graphical models, sampling, and variational inference all run on this algebra.
- **Ch 20**: the KL asymmetry decides GAN vs VAE failure modes.
slack-gif-creatoranthropics/skillsKnowledge and utilities for creating animated GIFs optimized for Slack. Provides constraints, validation tools, and animation concepts. Use when users request animated GIFs for Slack like "make me a GIF of X doing Y for Slack."Stars: 198kUpdated Claude CodeCodexVerifiedInstall
ponytailDietrichGebert/ponytailLazy senior dev mode for any coding task: the smallest change that fully solves it, and a reply a busy human understands in one read.Stars: 194kUpdated Claude CodeCodexVerifiedInstall
ponytail-debtDietrichGebert/ponytailHarvest every shortcut: comment into one debt ledger, so deferrals get tracked instead of forgotten. One-shot report.Stars: 194kUpdated Claude CodeCodexVerifiedInstall
ponytail-debtDietrichGebert/ponytailList every `shortcut:` comment (and older `ponytail:` ones) as a debt ledger. One-shot report, changes nothing. Use for "ponytail debt", "what did ponytail defer", "list the shortcuts", /ponytail-debt.Stars: 194kUpdated Claude CodeCodexVerifiedInstall