Skip to content

alirezarezvani/claude-skills / engineering/deep-learning-book/skills/deep-learning-book

deep-learning-book

Verified

Use when studying or applying Goodfellow, Bengio and Courville's Deep Learning, or asking if its advice still holds. Indexes all 20 chapters with a 2016-to-2026 delta, plans reading paths with reading_path_planner.py and diagnoses training failures with training_diagnostics.py.

FileKindSize
SKILL.mdskill md10.8 KB
assets/chapter_worksheet.mdmarkdown2.3 KB
assets/example_layer_spec.jsonconfig1.2 KB
assets/study_log_template.mdmarkdown2.0 KB
chapters/ch01-introduction.mdmarkdown3.1 KB
chapters/ch02-linear-algebra.mdmarkdown3.1 KB
chapters/ch03-probability-information-theory.mdmarkdown3.6 KB
chapters/ch04-numerical-computation.mdmarkdown3.0 KB
chapters/ch05-machine-learning-basics.mdmarkdown3.5 KB
chapters/ch06-deep-feedforward-networks.mdmarkdown3.2 KB
chapters/ch07-regularization.mdmarkdown3.6 KB
chapters/ch08-optimization.mdmarkdown3.5 KB
chapters/ch09-convolutional-networks.mdmarkdown3.4 KB
chapters/ch10-sequence-modeling.mdmarkdown3.5 KB
chapters/ch11-practical-methodology.mdmarkdown3.2 KB
chapters/ch12-applications.mdmarkdown3.0 KB
chapters/ch13-linear-factor-models.mdmarkdown3.1 KB
chapters/ch14-autoencoders.mdmarkdown3.3 KB
chapters/ch15-representation-learning.mdmarkdown3.5 KB
chapters/ch16-structured-probabilistic-models.mdmarkdown3.2 KB
chapters/ch17-monte-carlo-methods.mdmarkdown3.0 KB
chapters/ch18-partition-function.mdmarkdown3.5 KB
chapters/ch19-approximate-inference.mdmarkdown3.3 KB
chapters/ch20-deep-generative-models.mdmarkdown3.7 KB
cheatsheet.mdmarkdown3.2 KB
glossary.mdmarkdown5.9 KB
patterns.mdmarkdown4.7 KB
references/book_to_2026_delta.mdmarkdown6.4 KB
references/prerequisite_map.mdmarkdown4.9 KB
references/rights_and_use.mdmarkdown4.4 KB
references/study_method_canon.mdmarkdown5.3 KB
scripts/capacity_planner.pyscript14.4 KB
scripts/model_arithmetic.pyscript16.9 KB
scripts/reading_path_planner.pyscript19.7 KB
scripts/training_diagnostics.pyscript14.1 KB

chapters/ch19-approximate-inference.md

View on GitHub
# Chapter 19: Approximate Inference

**Source chapter (free, official):** https://www.deeplearningbook.org/contents/inference.html

## Core Idea

Computing p(h|x) — the posterior over latents — is intractable for most interesting models. The
fix is to reframe inference as optimization: choose a tractable family q, and maximize a lower
bound on the log-likelihood. The gap between the bound and the truth is exactly a KL divergence.

## Frameworks Introduced

- **Inference as optimization**: pick q(h) to maximize the **evidence lower bound (ELBO)**,
  L(q) = log p(x) − D_KL(q(h) ‖ p(h|x)). Maximizing L both fits the model and tightens the bound.
- **Expectation maximization (EM)**: alternate between setting q to the current posterior
  (E-step) and maximizing with respect to parameters (M-step).
- **MAP inference and sparse coding**: taking a point estimate of h is a degenerate q (a Dirac);
  sparse coding's inference step is exactly MAP inference.
- **Variational inference and mean field**: restrict q to a factorized family, q(h) = ∏ q(hᵢ),
  and derive fixed-point updates. Tractability is bought with an independence assumption.
- **Reverse-KL consequences**: the ELBO uses D(q‖p), which is mode-seeking — variational
  posteriors are characteristically **too narrow**, and underestimate uncertainty.
- **Learned approximate inference / amortization**: train a network to output q's parameters
  directly, replacing per-example optimization with one forward pass.

## Mental Models

- Read the ELBO as **"log-likelihood minus the cost of your approximation"** — improving q and
  improving the model are the same optimization.
- Expect **underestimated variance** from mean-field posteriors, and never report variational
  uncertainty as calibrated without checking.
- Treat amortization as the **encoder** of Chapter 14: the VAE is exactly this idea plus the
  reparameterization trick.

## Anti-patterns

- **Reporting a mean-field posterior's credible intervals as if they were exact.**
- **Blaming the model for a poor fit** that is actually a too-restrictive q — diagnose the bound
  before the model.

## What changed after 2016

Amortized variational inference became routine (VAEs and descendants); normalizing flows and
importance-weighted bounds (IWAE) gave tighter, more expressive posteriors than mean field. But
the strategic picture changed more: modern large generative models largely **avoid latent-variable
posteriors altogether** — autoregressive transformers have no posterior to infer, and diffusion
models use a fixed forward process, so their "inference" is trivial by construction. Variational
inference remains central in Bayesian deep learning and in structured latent-variable modelling.
**Confidence: high.**

## Key Takeaways

1. Write the ELBO down for any latent-variable model you train; it tells you what you are
   actually optimizing.
2. Assume a mean-field posterior is over-confident until proven otherwise.
3. Ask whether your problem needs a latent posterior at all — many modern designs are structured
   to avoid one.

## Connects To

- **Ch 13 / Ch 14**: the latent-variable models needing inference, and amortization.
- **Ch 20**: the VAE, assembled from this chapter plus the reparameterization trick.
- **Ch 3**: the KL asymmetry that dictates the narrow-posterior failure mode.
slack-gif-creatoranthropics/skillsKnowledge and utilities for creating animated GIFs optimized for Slack. Provides constraints, validation tools, and animation concepts. Use when users request animated GIFs for Slack like "make me a GIF of X doing Y for Slack."Stars: 198kUpdated Claude CodeCodexVerifiedInstall
ponytailDietrichGebert/ponytailLazy senior dev mode for any coding task: the smallest change that fully solves it, and a reply a busy human understands in one read.Stars: 194kUpdated Claude CodeCodexVerifiedInstall
ponytail-debtDietrichGebert/ponytailHarvest every shortcut: comment into one debt ledger, so deferrals get tracked instead of forgotten. One-shot report.Stars: 194kUpdated Claude CodeCodexVerifiedInstall
ponytail-debtDietrichGebert/ponytailList every `shortcut:` comment (and older `ponytail:` ones) as a debt ledger. One-shot report, changes nothing. Use for "ponytail debt", "what did ponytail defer", "list the shortcuts", /ponytail-debt.Stars: 194kUpdated Claude CodeCodexVerifiedInstall