michalstrnadel/source-to-skill / examples/neural-networks-3blue1brown
neural-networks-3blue1brown
VerifiedCourse skill distilled from 3Blue1Brown's "Neural networks" YouTube playlist (10 videos) - from a plain MNIST digit classifier (neurons, weights, biases, sigmoid/ReLU), through cost functions, gradient descent, stochastic gradient descent and backpropagation with its chain-rule calculus, to large language models and transformers (tokens, embeddings, softmax and temperature, attention heads with query/key/value, masking, MLP blocks storing facts, superposition, the full GPT-3 175B parameter tally), cross-entropy loss, KL divergence and distillation, and diffusion image/video models (CLIP, DDPM, DDIM, classifier-free guidance). Load when explaining or teaching how neural networks, backprop, transformers, attention, LLM training losses or diffusion models work, when someone needs an intuition-first mental model or a concrete number (e.g. GPT-3 dimensions), or asks "what does this 3Blue1Brown lesson say about X".
| File | Kind | Size |
|---|---|---|
| SKILL.md | skill md | 9.3 KB |
| cheatsheet.md | markdown | 5.8 KB |
| evals/evals.json | config | 3.8 KB |
| evals/last-run.json | config | 2.1 KB |
| lessons/01-what-is-a-neural-network.md | markdown | 4.5 KB |
| lessons/02-gradient-descent.md | markdown | 4.9 KB |
| lessons/03-backpropagation-intuition.md | markdown | 3.7 KB |
| lessons/04-backpropagation-calculus.md | markdown | 2.9 KB |
| lessons/05-llms-explained-briefly.md | markdown | 3.1 KB |
| lessons/06-transformers.md | markdown | 6.8 KB |
| lessons/07-attention.md | markdown | 5.8 KB |
| lessons/08-mlps-store-facts.md | markdown | 5.2 KB |
| lessons/09-cross-entropy.md | markdown | 6.5 KB |
| lessons/10-diffusion-models.md | markdown | 7.4 KB |
| source.json | config | 1.8 KB |
lessons/03-backpropagation-intuition.md
View on GitHubhttps://www.youtube.com/watch?v=Ilg3gGewQ5U
# Lesson 3 — Backpropagation, intuitively (Deep Learning Chapter 3)
Backpropagation is the algorithm for computing the cost gradient — "the core
algorithm behind how neural networks learn". This lesson explains it with no
formulas; lesson 4 gives the calculus.
## Key reframe: gradient components = sensitivities
- Each gradient component's magnitude says how sensitive the cost is to that
weight or bias.
- Example: components 3.2 and 0.1 → the cost is **32×** more sensitive to the
first weight. ([2:04](https://www.youtube.com/watch?v=Ilg3gGewQ5U&t=124s))
- The hard part is notation and index-chasing; each individual effect is
intuitive, there are just many layered on top of each other.
## One training example (an image of a 2) ([3:07](https://www.youtube.com/watch?v=Ilg3gGewQ5U&t=187s))
1. **Desired output nudges.** You cannot change activations directly, only
weights and biases — but record what you *wish* would happen: the "2"
neuron up, all others down, each nudge proportional to how far the value is
from its target (a neuron already near 0 needs little change). ([4:09](https://www.youtube.com/watch?v=Ilg3gGewQ5U&t=249s))
2. **Three levers for raising the "2" neuron's activation:** ([4:09](https://www.youtube.com/watch?v=Ilg3gGewQ5U&t=249s))
- increase its **bias**;
- increase its **weights** — most effective on connections from the
*brightest* previous neurons, since those weights are multiplied by large
activations;
- change the **previous layer's activations** — brighten neurons with
positive weights, dim those with negative weights, in proportion to the
weight size.
3. **Hebbian echo.** The biggest weight increases happen between neurons that
are most active and the ones we want more active — loosely, "neurons that
fire together wire together". The course flags this as a loose analogy only. ([5:10](https://www.youtube.com/watch?v=Ilg3gGewQ5U&t=310s))
4. **Propagate backwards.** Every output neuron has its own wishes for the
second-to-last layer. Sum them (weighted by connection weights and by how
much each output must change) → a list of desired nudges for that layer.
Recursively apply the same process to the weights and biases feeding it,
moving back through the network. ([7:12](https://www.youtube.com/watch?v=Ilg3gGewQ5U&t=432s))
## All training examples ([8:14](https://www.youtube.com/watch?v=Ilg3gGewQ5U&t=494s))
- Listening only to the "2" would push the net to call everything a 2. So run
backprop for **every** example and **average** the desired changes. That
average is (proportional to) the negative gradient.
## Stochastic gradient descent ([9:14](https://www.youtube.com/watch?v=Ilg3gGewQ5U&t=554s))
- Summing every example for every step is very slow. Instead: shuffle the
data, split into **mini-batches** (e.g. 100 examples), take one step per
mini-batch.
- Each step approximates the true gradient — not the most efficient downhill
direction, but a big computational speedup.
- Picture: a drunk man stumbling quickly downhill rather than a careful man
computing the exact slope before each slow step. ([10:17](https://www.youtube.com/watch?v=Ilg3gGewQ5U&t=617s))
- Repeating over all mini-batches converges to a local minimum = good
performance on the training data.
## Summary as stated
Backprop determines how a single example wants to nudge each weight and bias —
direction *and* relative proportion. A true gradient step averages this over
all examples; in practice you use mini-batches.
## Practical note ([11:17](https://www.youtube.com/watch?v=Ilg3gGewQ5U&t=677s))
These methods need **a lot of labeled training data**. MNIST makes digits an
easy example; in practice, getting labeled data is a common bottleneck.
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| reverse-engineer-anythingmorluto/reaReverse engineer native, managed, Electron/JavaScript, packaged, firmware, and browser targets with REA, and build agent environments (AgentEnv bundles) of web applications from a live instance or a capture, and of Linux desktop apps with a task from a request. Use shipped-artifact or requested runtime evidence to explain features, compare versions, decompile code, or guide a reconstruction. Skip REA for ordinary source-repository architecture analysis. | Stars: 472k | Updated | Claude CodeCodex | Verified | Install |
| diagnosing-superpowersobra/superpowersUse when a superpowers session went wrong and your human partner wants to know why — repeated work, ignored plans, stumbles, poor results, a skill that didn't fire, "it took too long", "why is it so expensive", "what is it doing" — or wants to build a bug report for the superpowers maintainers, for the current session or a past one identified by id or path, on any harness. | Stars: 323k | Updated | Claude CodeCodex | Verified | Install |
| dispatching-parallel-agentsobra/superpowersUse when facing 2+ independent tasks that can be worked on without shared state or sequential dependencies | Stars: 323k | Updated | Claude CodeCodex | Verified | Install |
| executing-plansobra/superpowersUse when executing an implementation plan in the current session as the implementer yourself — your human partner chose inline execution, or no subagent tool is available | Stars: 323k | Updated | Claude CodeCodex | Verified | Install |