Skip to content

alirezarezvani/claude-skills / engineering/deep-learning-book/skills/deep-learning-book

deep-learning-book

Verified

Use when studying or applying Goodfellow, Bengio and Courville's Deep Learning, or asking if its advice still holds. Indexes all 20 chapters with a 2016-to-2026 delta, plans reading paths with reading_path_planner.py and diagnoses training failures with training_diagnostics.py.

FileKindSize
SKILL.mdskill md10.8 KB
assets/chapter_worksheet.mdmarkdown2.3 KB
assets/example_layer_spec.jsonconfig1.2 KB
assets/study_log_template.mdmarkdown2.0 KB
chapters/ch01-introduction.mdmarkdown3.1 KB
chapters/ch02-linear-algebra.mdmarkdown3.1 KB
chapters/ch03-probability-information-theory.mdmarkdown3.6 KB
chapters/ch04-numerical-computation.mdmarkdown3.0 KB
chapters/ch05-machine-learning-basics.mdmarkdown3.5 KB
chapters/ch06-deep-feedforward-networks.mdmarkdown3.2 KB
chapters/ch07-regularization.mdmarkdown3.6 KB
chapters/ch08-optimization.mdmarkdown3.5 KB
chapters/ch09-convolutional-networks.mdmarkdown3.4 KB
chapters/ch10-sequence-modeling.mdmarkdown3.5 KB
chapters/ch11-practical-methodology.mdmarkdown3.2 KB
chapters/ch12-applications.mdmarkdown3.0 KB
chapters/ch13-linear-factor-models.mdmarkdown3.1 KB
chapters/ch14-autoencoders.mdmarkdown3.3 KB
chapters/ch15-representation-learning.mdmarkdown3.5 KB
chapters/ch16-structured-probabilistic-models.mdmarkdown3.2 KB
chapters/ch17-monte-carlo-methods.mdmarkdown3.0 KB
chapters/ch18-partition-function.mdmarkdown3.5 KB
chapters/ch19-approximate-inference.mdmarkdown3.3 KB
chapters/ch20-deep-generative-models.mdmarkdown3.7 KB
cheatsheet.mdmarkdown3.2 KB
glossary.mdmarkdown5.9 KB
patterns.mdmarkdown4.7 KB
references/book_to_2026_delta.mdmarkdown6.4 KB
references/prerequisite_map.mdmarkdown4.9 KB
references/rights_and_use.mdmarkdown4.4 KB
references/study_method_canon.mdmarkdown5.3 KB
scripts/capacity_planner.pyscript14.4 KB
scripts/model_arithmetic.pyscript16.9 KB
scripts/reading_path_planner.pyscript19.7 KB
scripts/training_diagnostics.pyscript14.1 KB

chapters/ch15-representation-learning.md

View on GitHub
# Chapter 15: Representation Learning

**Source chapter (free, official):** https://www.deeplearningbook.org/contents/representation.html

## Core Idea

What makes one representation better than another, and how do you get one without labels? The
chapter names the properties worth wanting — distributed, disentangled, capturing causal factors
— and the transfer mechanisms that let a representation earned on one task pay off on another.

## Frameworks Introduced

- **Greedy layer-wise unsupervised pretraining**: the historically important technique that
  restarted the field in 2006; the chapter is already candid that its value had shrunk by 2016.
- **Transfer learning and domain adaptation**: reuse a representation across tasks or
  distributions; **one-shot / zero-shot learning** as the limiting cases.
- **Distributed representations**: n binary features describe 2ⁿ regions, versus n regions for a
  one-hot/symbolic code. This exponential advantage is the central argument for learned features.
- **Disentangled factors**: separate underlying causes onto separate directions.
- **Exponential gains from depth**: deeper composition of distributed features again multiplies
  expressible structure.
- **Causal factors and semi-supervised learning**: unsupervised learning helps supervised learning
  exactly when p(x) and p(y|x) share structure — if the factors generating x include y's causes.
- **Regularization priors that define good representations**: smoothness, linearity, multiple
  explanatory factors, hierarchy, sparsity, simplicity of factor dependencies, shared factors
  across tasks, manifolds, temporal/spatial coherence.

## Mental Models

- Judge a representation by **what becomes linearly separable** in it — that is the operational
  version of "good features."
- Use the **shared-cause test** to predict whether unlabeled data will help: if p(x) tells you
  nothing about p(y|x), self-supervision will not rescue a supervised task.
- Read distributed-vs-symbolic as **the reason embeddings beat lookup tables**, and note that the
  advantage is combinatorial, not merely empirical.

## Anti-patterns

- **Expecting unsupervised pretraining to help unconditionally** — the chapter itself is careful
  here, and the 2016-era conclusion (it often does not, for large labeled datasets) was correct
  for the methods then available.
- **Claiming disentanglement without a metric or an intervention.**

## What changed after 2016

This chapter aged into relevance rather than out of it. Self-supervised learning became the
dominant paradigm — contrastive methods (SimCLR, MoCo, CLIP), masked prediction (BERT, MAE), and
next-token prediction at scale — vindicating the chapter's core bet while replacing every
specific method it lists. Two corrections: greedy layer-wise pretraining is now purely
historical, and unsupervised disentanglement was proven impossible without inductive bias
(Locatello et al. 2019). Linear-probe evaluation became the standard test of representation
quality. **Confidence: high.**

## Key Takeaways

1. Evaluate representations with linear probes and downstream transfer, not reconstruction.
2. Before investing in self-supervision, argue that p(x) and p(y|x) share causes.
3. Treat "disentangled" as a claim requiring an intervention-based test.

## Connects To

- **Ch 14**: autoencoders as one route to a representation.
- **Ch 1**: the promise made in the introduction, cashed out here.
- **references/book_to_2026_delta.md**: the self-supervised learning line.
slack-gif-creatoranthropics/skillsKnowledge and utilities for creating animated GIFs optimized for Slack. Provides constraints, validation tools, and animation concepts. Use when users request animated GIFs for Slack like "make me a GIF of X doing Y for Slack."Stars: 198kUpdated Claude CodeCodexVerifiedInstall
ponytailDietrichGebert/ponytailLazy senior dev mode for any coding task: the smallest change that fully solves it, and a reply a busy human understands in one read.Stars: 194kUpdated Claude CodeCodexVerifiedInstall
ponytail-debtDietrichGebert/ponytailHarvest every shortcut: comment into one debt ledger, so deferrals get tracked instead of forgotten. One-shot report.Stars: 194kUpdated Claude CodeCodexVerifiedInstall
ponytail-debtDietrichGebert/ponytailList every `shortcut:` comment (and older `ponytail:` ones) as a debt ledger. One-shot report, changes nothing. Use for "ponytail debt", "what did ponytail defer", "list the shortcuts", /ponytail-debt.Stars: 194kUpdated Claude CodeCodexVerifiedInstall