michalstrnadel/source-to-skill / examples/neural-networks-3blue1brown
neural-networks-3blue1brown
VerifiedCourse skill distilled from 3Blue1Brown's "Neural networks" YouTube playlist (10 videos) - from a plain MNIST digit classifier (neurons, weights, biases, sigmoid/ReLU), through cost functions, gradient descent, stochastic gradient descent and backpropagation with its chain-rule calculus, to large language models and transformers (tokens, embeddings, softmax and temperature, attention heads with query/key/value, masking, MLP blocks storing facts, superposition, the full GPT-3 175B parameter tally), cross-entropy loss, KL divergence and distillation, and diffusion image/video models (CLIP, DDPM, DDIM, classifier-free guidance). Load when explaining or teaching how neural networks, backprop, transformers, attention, LLM training losses or diffusion models work, when someone needs an intuition-first mental model or a concrete number (e.g. GPT-3 dimensions), or asks "what does this 3Blue1Brown lesson say about X".
| File | Kind | Size |
|---|---|---|
| SKILL.md | skill md | 9.3 KB |
| cheatsheet.md | markdown | 5.8 KB |
| evals/evals.json | config | 3.8 KB |
| evals/last-run.json | config | 2.1 KB |
| lessons/01-what-is-a-neural-network.md | markdown | 4.5 KB |
| lessons/02-gradient-descent.md | markdown | 4.9 KB |
| lessons/03-backpropagation-intuition.md | markdown | 3.7 KB |
| lessons/04-backpropagation-calculus.md | markdown | 2.9 KB |
| lessons/05-llms-explained-briefly.md | markdown | 3.1 KB |
| lessons/06-transformers.md | markdown | 6.8 KB |
| lessons/07-attention.md | markdown | 5.8 KB |
| lessons/08-mlps-store-facts.md | markdown | 5.2 KB |
| lessons/09-cross-entropy.md | markdown | 6.5 KB |
| lessons/10-diffusion-models.md | markdown | 7.4 KB |
| source.json | config | 1.8 KB |
lessons/10-diffusion-models.md
View on GitHubhttps://www.youtube.com/watch?v=iv-5mZ_9CPY
# Lesson 10 — But how do AI images and videos actually work? (guest video by Welch Labs)
Guest video by Stephen Welch (Welch Labs), commissioned during 3Blue1Brown's
paternity leave. Thesis: image/video models use **diffusion**, equivalent to
Brownian motion run backwards in high-dimensional space — and the physics
yields real algorithms.
## What a video model actually does (WAN 2.1, open source)
- Generation starts from a random-number call: a pure-noise video. ([1:01](https://www.youtube.com/watch?v=iv-5mZ_9CPY&t=61s))
- A **transformer** (same model type as LLMs) outputs a video; it is added to
the noise and fed back in; repeat (snapshots at 5, 10, 20, 30, 40, 50
iterations) until realistic video emerges.
- An empty prompt still yields a video (of a woman).
## Part 1 — CLIP (OpenAI, Feb 2021) ([3:08](https://www.youtube.com/watch?v=iv-5mZ_9CPY&t=188s))
- Trained on **400 million** image–caption pairs; two models (text, image),
each outputting a **512**-length vector.
- **Contrastive** objective (the "C"): in a batch, arrange image vectors as
columns and text vectors as rows; maximize similarity on the diagonal
(matching pairs) and minimize it off-diagonal. ([5:11](https://www.youtube.com/watch?v=iv-5mZ_9CPY&t=311s))
- Similarity = **cosine similarity** (cosine of the angle; 1 when aligned).
- The shared **latent/embedding space** supports concept arithmetic: (me with
hat) − (me without hat) best matches the word "hat" (similarity 0.165), then
"cap", "helmet". ([6:16](https://www.youtube.com/watch?v=iv-5mZ_9CPY&t=376s))
- Zero-shot classification: compare an image's vector with one caption per
label; pick the highest cosine similarity. ([7:16](https://www.youtube.com/watch?v=iv-5mZ_9CPY&t=436s))
- Limitation: CLIP only maps **into** the space; it cannot generate.
## Part 2 — Diffusion (DDPM, Berkeley 2020) ([8:20](https://www.youtube.com/watch?v=iv-5mZ_9CPY&t=500s))
- Core idea: add noise to training images step by step until destroyed; train
a network to reverse it.
- The naive version — predict step t−1 from step t, iterate — "really does not
work well"; virtually no modern model does that. Two surprises in DDPM: ([9:22](https://www.youtube.com/watch?v=iv-5mZ_9CPY&t=562s))
1. **Noise is added during generation too.** Removing that line from Stable
Diffusion 2 sampling turns "a tree in the desert" into a tiny, sad, blurry
tree. ([10:23](https://www.youtube.com/watch?v=iv-5mZ_9CPY&t=623s))
2. **The model predicts the total noise** ε added to the clean image x₀, not
one step's noise. ([10:23](https://www.youtube.com/watch?v=iv-5mZ_9CPY&t=623s))
### The vector-field view (2-D toy: images as points, data on a spiral) ([12:24](https://www.youtube.com/watch?v=iv-5mZ_9CPY&t=744s))
- Adding noise = a random walk = **Brownian motion**; the model must play
diffusion backwards. ([13:27](https://www.youtube.com/watch?v=iv-5mZ_9CPY&t=807s))
- Predicting the final step's noise is mathematically equivalent to predicting
total noise ÷ number of steps; predicting the vector back to x₀ directly
keeps the objective but **reduces training variance** → learns far more
efficiently. ([15:32](https://www.youtube.com/watch?v=iv-5mZ_9CPY&t=932s))
- The model learns, at each point, the direction back toward the data — a
**score function** pointing to more likely, less noisy data.
- **Condition on time t** (t = 1 at the 100th step, 0.99 at the 99th, ...):
coarse fields at large t, fine structure as t → 0. Essential in practice. A
sudden "phase change" near t ≈ 0.4: the field switches from pointing at the
spiral's center to pointing at the spiral itself. ([16:33](https://www.youtube.com/watch?v=iv-5mZ_9CPY&t=993s))
### Why sampling noise matters
- DDPM sampling: step along the learned field, then add scaled random noise
(shrinking over time). 64 steps; a 256-point cloud converges onto the spiral. ([17:38](https://www.youtube.com/watch?v=iv-5mZ_9CPY&t=1058s))
- Without the noise, all points rush to the center, then to one inner edge.
The model learns the **mean** of the (Gaussian, for small steps) reverse
distribution; to *sample* you must add zero-mean Gaussian noise. Without it
you land on averages — and **in image space, averages look blurry**. ([18:42](https://www.youtube.com/watch?v=iv-5mZ_9CPY&t=1122s))
## DDIM — faster, deterministic ([21:45](https://www.youtube.com/watch?v=iv-5mZ_9CPY&t=1305s))
- DDPM's many steps (each a full network pass) were compute-heavy.
- DDPM sampling is a **stochastic differential equation** (field term + random
term). Via the **Fokker–Planck equation**, a Google Brain team found an
**ordinary** differential equation (no randomness) giving the same final
**distribution** (not the same individual images). ([22:45](https://www.youtube.com/watch?v=iv-5mZ_9CPY&t=1365s))
- Result, **DDIM**: no noise during sampling, new step-size scaling (which
matters a lot), high quality in far fewer steps, **no retraining**. ([23:47](https://www.youtube.com/watch?v=iv-5mZ_9CPY&t=1427s))
- WAN uses a generalization of DDIM called **flow matching**. ([24:47](https://www.youtube.com/watch?v=iv-5mZ_9CPY&t=1487s))
## Part 3 — Steering with text
- Combine: diffusion can invert the CLIP image encoder; CLIP text vectors can
steer diffusion. OpenAI did this in 2022 (**unCLIP**, sold as **DALL·E 2**)
with remarkable prompt adherence. ([24:47](https://www.youtube.com/watch?v=iv-5mZ_9CPY&t=1487s))
- **Conditioning**: feed the text vector as an extra model input (via
cross-attention, adding/appending to the input, or several ways at once). ([26:53](https://www.youtube.com/watch?v=iv-5mZ_9CPY&t=1613s))
- Conditioning alone is not enough: Stable Diffusion (Heidelberg University,
open source, similar approach) conditioned only on text gives a desert with a
shadow but **no tree**. ([27:53](https://www.youtube.com/watch?v=iv-5mZ_9CPY&t=1673s))
### Classifier-free guidance
- Toy: spiral regions = people / dogs / cats; class-conditioned model fits
roughly but confuses classes — matching the overall data overpowers moving
toward a class. ([28:53](https://www.youtube.com/watch?v=iv-5mZ_9CPY&t=1733s))
- Train one model that sometimes gets **no** class info (drop it for a subset
of examples) → you get both an unconditional and a conditional vector field.
They agree at large t and diverge as t → 0. ([29:58](https://www.youtube.com/watch?v=iv-5mZ_9CPY&t=1798s))
- Guided direction: **(conditional − unconditional)**, amplified by a factor
**α**, replaces the conditional vector → tight fits per class. ([31:00](https://www.youtube.com/watch?v=iv-5mZ_9CPY&t=1860s))
- Stable Diffusion: a tiny tree appears around **α ≈ 2** and grows in size and
detail as α increases. Now essential in modern image/video models. ([33:03](https://www.youtube.com/watch?v=iv-5mZ_9CPY&t=1983s))
- **Negative prompts** (WAN): subtract the vector for an explicit list of
unwanted features (e.g. extra fingers, walking backwards — written in
Chinese) instead of the unconditioned output. Without it, scenes get
cartoonish and parts stop fitting together. ([33:03](https://www.youtube.com/watch?v=iv-5mZ_9CPY&t=1983s))
## Takeaway
Since DDPM (summer 2020) progress has been blistering. The striking part is
that independently trained pieces (a CLIP-style text encoder steering a
diffusion model) fit together at all, and that simple geometric intuitions
hold in such high dimensions.
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| reverse-engineer-anythingmorluto/reaReverse engineer native, managed, Electron/JavaScript, packaged, firmware, and browser targets with REA, and build agent environments (AgentEnv bundles) of web applications from a live instance or a capture, and of Linux desktop apps with a task from a request. Use shipped-artifact or requested runtime evidence to explain features, compare versions, decompile code, or guide a reconstruction. Skip REA for ordinary source-repository architecture analysis. | Stars: 472k | Updated | Claude CodeCodex | Verified | Install |
| diagnosing-superpowersobra/superpowersUse when a superpowers session went wrong and your human partner wants to know why — repeated work, ignored plans, stumbles, poor results, a skill that didn't fire, "it took too long", "why is it so expensive", "what is it doing" — or wants to build a bug report for the superpowers maintainers, for the current session or a past one identified by id or path, on any harness. | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |
| dispatching-parallel-agentsobra/superpowersUse when facing 2+ independent tasks that can be worked on without shared state or sequential dependencies | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |
| executing-plansobra/superpowersUse when executing an implementation plan in the current session as the implementer yourself — your human partner chose inline execution, or no subagent tool is available | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |