Skip to content

alirezarezvani/claude-skills / engineering/deep-learning-book/skills/deep-learning-book

deep-learning-book

Verified

Use when studying or applying Goodfellow, Bengio and Courville's Deep Learning, or asking if its advice still holds. Indexes all 20 chapters with a 2016-to-2026 delta, plans reading paths with reading_path_planner.py and diagnoses training failures with training_diagnostics.py.

FileKindSize
SKILL.mdskill md10.8 KB
assets/chapter_worksheet.mdmarkdown2.3 KB
assets/example_layer_spec.jsonconfig1.2 KB
assets/study_log_template.mdmarkdown2.0 KB
chapters/ch01-introduction.mdmarkdown3.1 KB
chapters/ch02-linear-algebra.mdmarkdown3.1 KB
chapters/ch03-probability-information-theory.mdmarkdown3.6 KB
chapters/ch04-numerical-computation.mdmarkdown3.0 KB
chapters/ch05-machine-learning-basics.mdmarkdown3.5 KB
chapters/ch06-deep-feedforward-networks.mdmarkdown3.2 KB
chapters/ch07-regularization.mdmarkdown3.6 KB
chapters/ch08-optimization.mdmarkdown3.5 KB
chapters/ch09-convolutional-networks.mdmarkdown3.4 KB
chapters/ch10-sequence-modeling.mdmarkdown3.5 KB
chapters/ch11-practical-methodology.mdmarkdown3.2 KB
chapters/ch12-applications.mdmarkdown3.0 KB
chapters/ch13-linear-factor-models.mdmarkdown3.1 KB
chapters/ch14-autoencoders.mdmarkdown3.3 KB
chapters/ch15-representation-learning.mdmarkdown3.5 KB
chapters/ch16-structured-probabilistic-models.mdmarkdown3.2 KB
chapters/ch17-monte-carlo-methods.mdmarkdown3.0 KB
chapters/ch18-partition-function.mdmarkdown3.5 KB
chapters/ch19-approximate-inference.mdmarkdown3.3 KB
chapters/ch20-deep-generative-models.mdmarkdown3.7 KB
cheatsheet.mdmarkdown3.2 KB
glossary.mdmarkdown5.9 KB
patterns.mdmarkdown4.7 KB
references/book_to_2026_delta.mdmarkdown6.4 KB
references/prerequisite_map.mdmarkdown4.9 KB
references/rights_and_use.mdmarkdown4.4 KB
references/study_method_canon.mdmarkdown5.3 KB
scripts/capacity_planner.pyscript14.4 KB
scripts/model_arithmetic.pyscript16.9 KB
scripts/reading_path_planner.pyscript19.7 KB
scripts/training_diagnostics.pyscript14.1 KB

chapters/ch14-autoencoders.md

View on GitHub
# Chapter 14: Autoencoders

**Source chapter (free, official):** https://www.deeplearningbook.org/contents/autoencoders.html

## Core Idea

An autoencoder learns to copy its input imperfectly, and the constraint that prevents perfect
copying is where the learning happens. Undercompleteness, sparsity, denoising and contraction are
four different ways to impose that constraint — each defines a different notion of "useful".

## Frameworks Introduced

- **Undercomplete autoencoder**: bottleneck smaller than the input. With linear units and MSE it
  recovers PCA's subspace.
- **Regularized autoencoders**: capacity can exceed the input dimension as long as something else
  prevents identity — this is the chapter's key move.
- **Sparse autoencoder**: penalize code activation; interpretable as a latent prior (Ch 13).
- **Denoising autoencoder (DAE)**: corrupt the input, reconstruct the clean version. The learned
  map estimates the *score* — it points back toward the data manifold, which is the direct
  ancestor of score-based diffusion models.
- **Contractive autoencoder (CAE)**: penalize the Jacobian norm of the encoder, so the
  representation resists input perturbation except along the manifold.
- **Manifold learning view**: the encoder is sensitive along tangent directions of the data
  manifold and insensitive orthogonal to it.
- **Stochastic encoders and decoders**: autoencoders as p(h|x) and p(x|h).
- **Predictive sparse decomposition**; **applications**: dimensionality reduction, semantic
  hashing, pretraining.

## Mental Models

- Ask "**what stops it from learning the identity?**" — the answer names the inductive bias, and
  a model with no answer learns nothing useful.
- Read a DAE's learned vector field as **pointing uphill in density**. Once you see that,
  diffusion models are the same idea run at many noise levels.
- Treat the encoder as **amortized inference**: it replaces the per-example optimization of
  sparse coding with one forward pass.

## Anti-patterns

- **An overcomplete autoencoder with no regularizer** — it can and will learn a copy.
- **Judging representation quality by reconstruction error**: low reconstruction error can mean
  the code memorized the input, which is the opposite of useful.

## What changed after 2016

The denoising idea became the foundation of modern generative modelling: denoising score matching
(Vincent 2011) → score-based models (Song & Ermon 2019) → DDPM (Ho et al. 2020) → the diffusion
family. Greedy layerwise autoencoder pretraining, presented here as a live technique, disappeared
— displaced first by better initialization/normalization and then by large-scale supervised and
self-supervised pretraining. The VAE overshadowed the deterministic autoencoders for generation,
while masked autoencoders (He et al. 2021) revived the corruption idea for vision pretraining.
**Confidence: high.**

## Key Takeaways

1. Always name the constraint that prevents identity before training an autoencoder.
2. Do not use reconstruction error as a representation-quality metric; evaluate downstream.
3. Learn the denoising/score connection here — it is the cheapest on-ramp to diffusion models.

## Connects To

- **Ch 13**: sparse coding, amortized.
- **Ch 15**: what a "good" representation means.
- **Ch 20**: VAEs, and the diffusion line that grew from denoising.
slack-gif-creatoranthropics/skillsKnowledge and utilities for creating animated GIFs optimized for Slack. Provides constraints, validation tools, and animation concepts. Use when users request animated GIFs for Slack like "make me a GIF of X doing Y for Slack."Stars: 198kUpdated Claude CodeCodexVerifiedInstall
ponytailDietrichGebert/ponytailLazy senior dev mode for any coding task: the smallest change that fully solves it, and a reply a busy human understands in one read.Stars: 194kUpdated Claude CodeCodexVerifiedInstall
ponytail-debtDietrichGebert/ponytailHarvest every shortcut: comment into one debt ledger, so deferrals get tracked instead of forgotten. One-shot report.Stars: 194kUpdated Claude CodeCodexVerifiedInstall
ponytail-debtDietrichGebert/ponytailList every `shortcut:` comment (and older `ponytail:` ones) as a debt ledger. One-shot report, changes nothing. Use for "ponytail debt", "what did ponytail defer", "list the shortcuts", /ponytail-debt.Stars: 194kUpdated Claude CodeCodexVerifiedInstall