Skip to content

alirezarezvani/claude-skills / engineering/deep-learning-book/skills/deep-learning-book

deep-learning-book

Verified

Use when studying or applying Goodfellow, Bengio and Courville's Deep Learning, or asking if its advice still holds. Indexes all 20 chapters with a 2016-to-2026 delta, plans reading paths with reading_path_planner.py and diagnoses training failures with training_diagnostics.py.

FileKindSize
SKILL.mdskill md10.8 KB
assets/chapter_worksheet.mdmarkdown2.3 KB
assets/example_layer_spec.jsonconfig1.2 KB
assets/study_log_template.mdmarkdown2.0 KB
chapters/ch01-introduction.mdmarkdown3.1 KB
chapters/ch02-linear-algebra.mdmarkdown3.1 KB
chapters/ch03-probability-information-theory.mdmarkdown3.6 KB
chapters/ch04-numerical-computation.mdmarkdown3.0 KB
chapters/ch05-machine-learning-basics.mdmarkdown3.5 KB
chapters/ch06-deep-feedforward-networks.mdmarkdown3.2 KB
chapters/ch07-regularization.mdmarkdown3.6 KB
chapters/ch08-optimization.mdmarkdown3.5 KB
chapters/ch09-convolutional-networks.mdmarkdown3.4 KB
chapters/ch10-sequence-modeling.mdmarkdown3.5 KB
chapters/ch11-practical-methodology.mdmarkdown3.2 KB
chapters/ch12-applications.mdmarkdown3.0 KB
chapters/ch13-linear-factor-models.mdmarkdown3.1 KB
chapters/ch14-autoencoders.mdmarkdown3.3 KB
chapters/ch15-representation-learning.mdmarkdown3.5 KB
chapters/ch16-structured-probabilistic-models.mdmarkdown3.2 KB
chapters/ch17-monte-carlo-methods.mdmarkdown3.0 KB
chapters/ch18-partition-function.mdmarkdown3.5 KB
chapters/ch19-approximate-inference.mdmarkdown3.3 KB
chapters/ch20-deep-generative-models.mdmarkdown3.7 KB
cheatsheet.mdmarkdown3.2 KB
glossary.mdmarkdown5.9 KB
patterns.mdmarkdown4.7 KB
references/book_to_2026_delta.mdmarkdown6.4 KB
references/prerequisite_map.mdmarkdown4.9 KB
references/rights_and_use.mdmarkdown4.4 KB
references/study_method_canon.mdmarkdown5.3 KB
scripts/capacity_planner.pyscript14.4 KB
scripts/model_arithmetic.pyscript16.9 KB
scripts/reading_path_planner.pyscript19.7 KB
scripts/training_diagnostics.pyscript14.1 KB

chapters/ch10-sequence-modeling.md

View on GitHub
# Chapter 10: Sequence Modeling — Recurrent and Recursive Nets

**Source chapter (free, official):** https://www.deeplearningbook.org/contents/rnn.html

## Core Idea

Recurrence shares parameters across time the way convolution shares them across space, which lets
one model handle variable-length sequences. The price is that gradients must traverse many
multiplicative steps, producing the vanishing/exploding gradient problem — and gated
architectures exist to pay it.

## Frameworks Introduced

- **Unfolding the computational graph**: a recurrent definition becomes a deep feedforward graph
  with tied weights.
- **BPTT (backpropagation through time)** and **truncated BPTT**.
- **Teacher forcing**: train on ground-truth previous tokens; note the train/inference mismatch
  (exposure bias) it creates.
- **Vanishing and exploding gradients**: repeated multiplication by the recurrent Jacobian; the
  spectral radius decides which.
- **Gradient clipping** for the exploding half; **gating** for the vanishing half.
- **LSTM** (input/forget/output gates plus a cell with an additive path) and **GRU** (a
  two-gate simplification). The additive cell path is the mechanism — it makes the gradient path
  through time approximately linear.
- **Bidirectional RNNs**; **encoder–decoder / sequence-to-sequence** with a fixed-size context.
- **Attention** appears here as the fix for the encoder–decoder bottleneck.
- **Deep RNNs, recursive (tree-structured) nets, echo state networks, leaky units, skip
  connections through time.**

## Mental Models

- Read gating as **learned, data-dependent memory management**: the forget gate decides how long
  the additive highway stays open.
- Treat the fixed-size context vector in vanilla seq2seq as a **bottleneck** — the whole reason
  attention was invented.
- Expect **exposure bias** whenever you teacher-force: the model has never seen its own mistakes
  during training.

## Anti-patterns

- **Reaching for an RNN by default in 2026** for a task where a transformer is the standard
  baseline and the sequence fits in context.
- **Training a long-sequence RNN without gradient clipping.**
- **Ignoring the train/inference mismatch** in autoregressive generation.

## What changed after 2016

**The largest single delta in the book.** Attention, described here as an enhancement to
recurrent seq2seq, became the whole architecture with *Attention Is All You Need* (Vaswani et al.
2017) — one year after publication. Transformers replaced RNNs for essentially all large-scale
sequence work: parallel training over sequence positions, direct O(1) path length between any two
tokens, and much better scaling. What survived: gradient clipping, teacher forcing, exposure bias,
and the vanishing/exploding analysis. What returned: linear-time recurrent architectures
(S4/Mamba-style state-space models, 2021–2024) as a long-context alternative, which makes this
chapter's material relevant again rather than obsolete. **Confidence: high.**

**Read this chapter for the gradient-flow analysis, not for the architecture recommendation.**

## Key Takeaways

1. Learn the vanishing/exploding analysis here — it explains residual connections, LayerNorm
   placement, and state-space models alike.
2. Clip gradients whenever a recurrence is in the graph.
3. Treat this chapter's architecture advice as historical; treat its diagnosis as current.

## Connects To

- **Ch 8**: the same optimization pathologies, in the time dimension.
- **Ch 12**: NLP and speech applications built on these.
- **references/book_to_2026_delta.md**: the transformer displacement, in detail.
slack-gif-creatoranthropics/skillsKnowledge and utilities for creating animated GIFs optimized for Slack. Provides constraints, validation tools, and animation concepts. Use when users request animated GIFs for Slack like "make me a GIF of X doing Y for Slack."Stars: 198kUpdated Claude CodeCodexVerifiedInstall
ponytailDietrichGebert/ponytailLazy senior dev mode for any coding task: the smallest change that fully solves it, and a reply a busy human understands in one read.Stars: 194kUpdated Claude CodeCodexVerifiedInstall
ponytail-debtDietrichGebert/ponytailHarvest every shortcut: comment into one debt ledger, so deferrals get tracked instead of forgotten. One-shot report.Stars: 194kUpdated Claude CodeCodexVerifiedInstall
ponytail-debtDietrichGebert/ponytailList every `shortcut:` comment (and older `ponytail:` ones) as a debt ledger. One-shot report, changes nothing. Use for "ponytail debt", "what did ponytail defer", "list the shortcuts", /ponytail-debt.Stars: 194kUpdated Claude CodeCodexVerifiedInstall