michalstrnadel/source-to-skill / examples/neural-networks-3blue1brown
neural-networks-3blue1brown
VerifiedCourse skill distilled from 3Blue1Brown's "Neural networks" YouTube playlist (10 videos) - from a plain MNIST digit classifier (neurons, weights, biases, sigmoid/ReLU), through cost functions, gradient descent, stochastic gradient descent and backpropagation with its chain-rule calculus, to large language models and transformers (tokens, embeddings, softmax and temperature, attention heads with query/key/value, masking, MLP blocks storing facts, superposition, the full GPT-3 175B parameter tally), cross-entropy loss, KL divergence and distillation, and diffusion image/video models (CLIP, DDPM, DDIM, classifier-free guidance). Load when explaining or teaching how neural networks, backprop, transformers, attention, LLM training losses or diffusion models work, when someone needs an intuition-first mental model or a concrete number (e.g. GPT-3 dimensions), or asks "what does this 3Blue1Brown lesson say about X".
| File | Kind | Size |
|---|---|---|
| SKILL.md | skill md | 9.3 KB |
| cheatsheet.md | markdown | 5.8 KB |
| evals/evals.json | config | 3.8 KB |
| evals/last-run.json | config | 2.1 KB |
| lessons/01-what-is-a-neural-network.md | markdown | 4.5 KB |
| lessons/02-gradient-descent.md | markdown | 4.9 KB |
| lessons/03-backpropagation-intuition.md | markdown | 3.7 KB |
| lessons/04-backpropagation-calculus.md | markdown | 2.9 KB |
| lessons/05-llms-explained-briefly.md | markdown | 3.1 KB |
| lessons/06-transformers.md | markdown | 6.8 KB |
| lessons/07-attention.md | markdown | 5.8 KB |
| lessons/08-mlps-store-facts.md | markdown | 5.2 KB |
| lessons/09-cross-entropy.md | markdown | 6.5 KB |
| lessons/10-diffusion-models.md | markdown | 7.4 KB |
| source.json | config | 1.8 KB |
lessons/04-backpropagation-calculus.md
View on GitHubhttps://www.youtube.com/watch?v=tIeHLnjs5U8
# Lesson 4 — Backpropagation calculus (Deep Learning Chapter 4)
Assumes lesson 3. Goal: show how ML practitioners think about the chain rule
in networks. Expect confusion — pause and ponder.
## One neuron per layer ([1:01](https://www.youtube.com/watch?v=tIeHLnjs5U8&t=61s))
Network with three weights and three biases; focus on the last two neurons.
- Notation (superscripts are layer indices, not exponents):
- `a(L)` last activation, `a(L-1)` previous activation, `y` desired value.
- `z(L) = w(L)·a(L-1) + b(L)` (the weighted sum, named for convenience). ([2:05](https://www.youtube.com/watch?v=tIeHLnjs5U8&t=125s))
- `a(L) = σ(z(L))` (sigmoid, ReLU, ...).
- Cost of one example: `C0 = (a(L) − y)²`.
- Dependency chain: `w(L), a(L-1), b(L) → z(L) → a(L) → C0`.
### Chain rule ([3:08](https://www.youtube.com/watch?v=tIeHLnjs5U8&t=188s))
`∂C0/∂w(L) = ∂z(L)/∂w(L) · ∂a(L)/∂z(L) · ∂C0/∂a(L)`
- `∂C0/∂a(L) = 2(a(L) − y)` — proportional to the error, so a badly-wrong
output makes small changes matter a lot.
- `∂a(L)/∂z(L) = σ'(z(L))` — derivative of the chosen nonlinearity.
- `∂z(L)/∂w(L) = a(L-1)` — a weight's effect depends on how strong the previous
neuron is (the "fire together, wire together" idea).
### Other partials ([5:16](https://www.youtube.com/watch?v=tIeHLnjs5U8&t=316s))
- Bias: swap the first factor for `∂z/∂b = 1`.
- Previous activation: `∂z(L)/∂a(L-1) = w(L)`. You can't set `a(L-1)`
directly, but tracking its sensitivity lets you iterate the same chain rule
backwards to earlier weights and biases — the "propagating backwards". ([6:19](https://www.youtube.com/watch?v=tIeHLnjs5U8&t=379s))
### Averaging ([5:16](https://www.youtube.com/watch?v=tIeHLnjs5U8&t=316s))
The full cost averages over training examples, so its derivative averages
these per-example expressions. Each such derivative is one component of the
gradient vector.
## Many neurons per layer ([6:19](https://www.youtube.com/watch?v=tIeHLnjs5U8&t=379s))
Only "a few more indices":
- `k` indexes layer L−1, `j` indexes layer L; activations `a(L)_j`.
- Cost: `C0 = Σ_j (a(L)_j − y_j)²`.
- Weight from neuron k to neuron j: `w(L)_jk` (index order matches the weight
matrix from lesson 1).
- `z(L)_j = Σ_k w(L)_jk a(L-1)_k + b(L)_j`, `a(L)_j = σ(z(L)_j)`.
- `∂C/∂w(L)_jk` has essentially the same chain-rule form.
- **What changes:** `∂C/∂a(L-1)_k` must **sum over all paths** — neuron k
influences every `a(L)_j`, each of which affects the cost. ([8:23](https://www.youtube.com/watch?v=tIeHLnjs5U8&t=503s))
Once you know the cost's sensitivity to the second-to-last layer's
activations, repeat for the weights and biases feeding that layer. These
chain-rule expressions are the gradient components gradient descent uses. ([9:28](https://www.youtube.com/watch?v=tIeHLnjs5U8&t=568s))
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| reverse-engineer-anythingmorluto/reaReverse engineer native, managed, Electron/JavaScript, packaged, firmware, and browser targets with REA, and build agent environments (AgentEnv bundles) of web applications from a live instance or a capture, and of Linux desktop apps with a task from a request. Use shipped-artifact or requested runtime evidence to explain features, compare versions, decompile code, or guide a reconstruction. Skip REA for ordinary source-repository architecture analysis. | Stars: 472k | Updated | Claude CodeCodex | Verified | Install |
| diagnosing-superpowersobra/superpowersUse when a superpowers session went wrong and your human partner wants to know why — repeated work, ignored plans, stumbles, poor results, a skill that didn't fire, "it took too long", "why is it so expensive", "what is it doing" — or wants to build a bug report for the superpowers maintainers, for the current session or a past one identified by id or path, on any harness. | Stars: 323k | Updated | Claude CodeCodex | Verified | Install |
| dispatching-parallel-agentsobra/superpowersUse when facing 2+ independent tasks that can be worked on without shared state or sequential dependencies | Stars: 323k | Updated | Claude CodeCodex | Verified | Install |
| executing-plansobra/superpowersUse when executing an implementation plan in the current session as the implementer yourself — your human partner chose inline execution, or no subagent tool is available | Stars: 323k | Updated | Claude CodeCodex | Verified | Install |