Skip to content

michalstrnadel/source-to-skill / examples/neural-networks-3blue1brown

neural-networks-3blue1brown

Verified

Course skill distilled from 3Blue1Brown's "Neural networks" YouTube playlist (10 videos) - from a plain MNIST digit classifier (neurons, weights, biases, sigmoid/ReLU), through cost functions, gradient descent, stochastic gradient descent and backpropagation with its chain-rule calculus, to large language models and transformers (tokens, embeddings, softmax and temperature, attention heads with query/key/value, masking, MLP blocks storing facts, superposition, the full GPT-3 175B parameter tally), cross-entropy loss, KL divergence and distillation, and diffusion image/video models (CLIP, DDPM, DDIM, classifier-free guidance). Load when explaining or teaching how neural networks, backprop, transformers, attention, LLM training losses or diffusion models work, when someone needs an intuition-first mental model or a concrete number (e.g. GPT-3 dimensions), or asks "what does this 3Blue1Brown lesson say about X".

lessons/04-backpropagation-calculus.md

View on GitHub
https://www.youtube.com/watch?v=tIeHLnjs5U8

# Lesson 4 — Backpropagation calculus (Deep Learning Chapter 4)

Assumes lesson 3. Goal: show how ML practitioners think about the chain rule
in networks. Expect confusion — pause and ponder.

## One neuron per layer ([1:01](https://www.youtube.com/watch?v=tIeHLnjs5U8&t=61s))

Network with three weights and three biases; focus on the last two neurons.

- Notation (superscripts are layer indices, not exponents):
  - `a(L)` last activation, `a(L-1)` previous activation, `y` desired value.
  - `z(L) = w(L)·a(L-1) + b(L)` (the weighted sum, named for convenience). ([2:05](https://www.youtube.com/watch?v=tIeHLnjs5U8&t=125s))
  - `a(L) = σ(z(L))` (sigmoid, ReLU, ...).
  - Cost of one example: `C0 = (a(L) − y)²`.
- Dependency chain: `w(L), a(L-1), b(L) → z(L) → a(L) → C0`.

### Chain rule ([3:08](https://www.youtube.com/watch?v=tIeHLnjs5U8&t=188s))

`∂C0/∂w(L) = ∂z(L)/∂w(L) · ∂a(L)/∂z(L) · ∂C0/∂a(L)`

- `∂C0/∂a(L) = 2(a(L) − y)` — proportional to the error, so a badly-wrong
  output makes small changes matter a lot.
- `∂a(L)/∂z(L) = σ'(z(L))` — derivative of the chosen nonlinearity.
- `∂z(L)/∂w(L) = a(L-1)` — a weight's effect depends on how strong the previous
  neuron is (the "fire together, wire together" idea).

### Other partials ([5:16](https://www.youtube.com/watch?v=tIeHLnjs5U8&t=316s))

- Bias: swap the first factor for `∂z/∂b = 1`.
- Previous activation: `∂z(L)/∂a(L-1) = w(L)`. You can't set `a(L-1)`
  directly, but tracking its sensitivity lets you iterate the same chain rule
  backwards to earlier weights and biases — the "propagating backwards". ([6:19](https://www.youtube.com/watch?v=tIeHLnjs5U8&t=379s))

### Averaging ([5:16](https://www.youtube.com/watch?v=tIeHLnjs5U8&t=316s))

The full cost averages over training examples, so its derivative averages
these per-example expressions. Each such derivative is one component of the
gradient vector.

## Many neurons per layer ([6:19](https://www.youtube.com/watch?v=tIeHLnjs5U8&t=379s))

Only "a few more indices":

- `k` indexes layer L−1, `j` indexes layer L; activations `a(L)_j`.
- Cost: `C0 = Σ_j (a(L)_j − y_j)²`.
- Weight from neuron k to neuron j: `w(L)_jk` (index order matches the weight
  matrix from lesson 1).
- `z(L)_j = Σ_k w(L)_jk a(L-1)_k + b(L)_j`, `a(L)_j = σ(z(L)_j)`.
- `∂C/∂w(L)_jk` has essentially the same chain-rule form.
- **What changes:** `∂C/∂a(L-1)_k` must **sum over all paths** — neuron k
  influences every `a(L)_j`, each of which affects the cost. ([8:23](https://www.youtube.com/watch?v=tIeHLnjs5U8&t=503s))

Once you know the cost's sensitivity to the second-to-last layer's
activations, repeat for the weights and biases feeding that layer. These
chain-rule expressions are the gradient components gradient descent uses. ([9:28](https://www.youtube.com/watch?v=tIeHLnjs5U8&t=568s))
reverse-engineer-anythingmorluto/reaReverse engineer native, managed, Electron/JavaScript, packaged, firmware, and browser targets with REA, and build agent environments (AgentEnv bundles) of web applications from a live instance or a capture, and of Linux desktop apps with a task from a request. Use shipped-artifact or requested runtime evidence to explain features, compare versions, decompile code, or guide a reconstruction. Skip REA for ordinary source-repository architecture analysis.Stars: 472kUpdated Claude CodeCodexVerifiedInstall
diagnosing-superpowersobra/superpowersUse when a superpowers session went wrong and your human partner wants to know why — repeated work, ignored plans, stumbles, poor results, a skill that didn't fire, "it took too long", "why is it so expensive", "what is it doing" — or wants to build a bug report for the superpowers maintainers, for the current session or a past one identified by id or path, on any harness.Stars: 323kUpdated Claude CodeCodexVerifiedInstall
dispatching-parallel-agentsobra/superpowersUse when facing 2+ independent tasks that can be worked on without shared state or sequential dependenciesStars: 323kUpdated Claude CodeCodexVerifiedInstall
executing-plansobra/superpowersUse when executing an implementation plan in the current session as the implementer yourself — your human partner chose inline execution, or no subagent tool is availableStars: 323kUpdated Claude CodeCodexVerifiedInstall