Skip to content

michalstrnadel/source-to-skill / examples/neural-networks-3blue1brown

neural-networks-3blue1brown

Verified

Course skill distilled from 3Blue1Brown's "Neural networks" YouTube playlist (10 videos) - from a plain MNIST digit classifier (neurons, weights, biases, sigmoid/ReLU), through cost functions, gradient descent, stochastic gradient descent and backpropagation with its chain-rule calculus, to large language models and transformers (tokens, embeddings, softmax and temperature, attention heads with query/key/value, masking, MLP blocks storing facts, superposition, the full GPT-3 175B parameter tally), cross-entropy loss, KL divergence and distillation, and diffusion image/video models (CLIP, DDPM, DDIM, classifier-free guidance). Load when explaining or teaching how neural networks, backprop, transformers, attention, LLM training losses or diffusion models work, when someone needs an intuition-first mental model or a concrete number (e.g. GPT-3 dimensions), or asks "what does this 3Blue1Brown lesson say about X".

Neural networks — 3Blue1Brown (course skill)

Source: https://www.youtube.com/playlist?list=PLZHQObOWTQDNU6R1_67000Dx_ZCJB-3pi (channel: 3Blue1Brown, 10 videos). Lesson 10 is a guest video by Welch Labs; lesson 9 is "Part 2" of a separate Compression is Intelligence series (its Part 1 is not in this playlist). Lesson files open with the video URL and link key moments as &t=<N>s deep links (timestamps from ~1-minute caption markers, so each link lands at or shortly before the point described).

What the course teaches, in order

  1. Structure (L1) — a network is just a function: 784 pixel inputs → 10 digit outputs, built from weighted sums + bias + a squishing nonlinearity, compactly a' = σ(W a + b).
  2. Learning = minimizing a cost (L2) — define a cost over all training data, follow the negative gradient downhill.
  3. Backpropagation (L3 intuition, L4 calculus) — how one training example "wants" every weight and bias nudged; average those wishes; use mini-batches (SGD) for speed.
  4. LLMs in brief (L5) — next-word prediction, pre-training, RLHF, GPUs and why transformers parallelize.
  5. Transformers (L6) — tokens → embeddings → alternating attention/MLP blocks → unembedding → softmax; deep-learning's "format rules".
  6. Attention (L7) — query/key dot products → softmax'd, masked attention pattern → value vectors added to embeddings; multi-head.
  7. MLPs store facts (L8) — up-projection + ReLU acts like AND-gates on feature directions; down-projection adds the associated fact; superposition.
  8. Cross-entropy (L9) — from compression ("bits wasted by the wrong code") to the LLM loss; why the log is forced; distillation; KL divergence.
  9. Diffusion (L10) — CLIP's shared text/image space, DDPM as reversed Brownian motion, DDIM, conditioning and classifier-free guidance.

Core mental models (front-loaded)

  • A neuron holds a number; a network is a function. Each neuron's activation = nonlinearity(weighted sum of previous layer + bias). Weights pick the pattern; the bias sets how high the sum must be before the neuron is meaningfully active. The 784-16-16-10 MNIST net has ~13,000 parameters.
  • Learning is calculus, not magic. Cost of one example = sum of squared differences between output and target; total cost = average over all examples. The negative gradient tells each parameter which way to move and how much it matters (relative magnitudes = "bang for your buck").
  • Smoothness is required. Gradient descent needs a smooth cost, which is why artificial neurons have continuous activations instead of on/off.
  • Backprop = recursively summed wishes. Output errors set desired nudges; each neuron's wish is split across bias, weights (proportional to the previous activation — "fire together, wire together") and previous-layer activations (proportional to the weights); wishes from all output neurons are summed, then the process repeats one layer back.
  • Chain rule per weight: ∂C/∂w(L) = a(L-1) · σ'(z(L)) · 2(a(L) − y); for the bias the first factor is 1; for the previous activation it is w(L), and with many neurons you sum over every path to the cost.
  • Deep learning's format: inputs and every layer are arrays of real numbers (tensors); parameters ("weights") touch data only through weighted sums (matrix-vector products); nonlinearities carry no parameters. Keep weights (the learned "brains") distinct from the data flowing through.
  • Directions carry meaning. Embedding differences encode concepts (woman − man ≈ a gender direction; cats − cat ≈ plurality). Dot products measure alignment: positive = similar, 0 = perpendicular, negative = opposite.
  • Attention moves information between positions; MLPs don't. Attention lets vectors update each other by context; the MLP processes every vector independently and in parallel and holds ~2/3 of GPT-3's parameters.
  • High dimensions hold far more "nearly perpendicular" features than dimensions (Johnson–Lindenstrauss), so features likely live in superposition across neurons — one reason models are hard to interpret and scale well.
  • Cross-entropy is the forced choice of loss. If you want the average loss minimized exactly when the model's distribution matches the data's, the per-token loss must be −log q. Minimum value = entropy of the data; the gap is the KL divergence.
  • Diffusion models learn a time-conditioned vector field pointing from noisy data back toward the data distribution; sampling follows it backward.

GPT-3 by the numbers (from L5–L8)

  • Vocabulary 50,257 tokens; embedding dimension 12,288; context 2,048 tokens.
  • 175B weights in just under 28,000 matrices, in 8 categories.
  • Embedding W_E ≈ 617M; unembedding W_U ≈ 617M.
  • Attention: key/query space 128-dim; ~6.3M params per head; 96 heads per block → ~600M per block; 96 layers → just under 58B (about 1/3 of total).
  • MLP: hidden size just under 50,000 (exactly 4 × 12,288); up + down ≈ 1.2B per block; 96 blocks → ~116B (about 2/3 of total).
  • Normalization and bias parameters: a trivial share.
  • Training compute: at 1 billion operations/second, training the largest models would take well over 100 million years (L5); a human reading GPT-3's training text 24/7 would need over 2,600 years.

Lesson index

#LessonOne-line takeawayFile
1But what is a neural network? (Ch. 1)Layers of weighted sums + bias + sigmoid; the whole net is a 13k-parameter functionlessons/01-what-is-a-neural-network.md
2Gradient descent, how neural networks learn (Ch. 2)Learning = minimizing average cost along the negative gradient; ~96–98% on MNIST, but hidden layers don't find "edges"lessons/02-gradient-descent.md
3Backpropagation, intuitively (Ch. 3)Each example's desired nudges propagate backward and are averaged; mini-batches give SGDlessons/03-backpropagation-intuition.md
4Backpropagation calculus (Ch. 4)Chain rule through z → a → C; sum over paths for multi-neuron layerslessons/04-backpropagation-calculus.md
5Large Language Models explained brieflyNext-word probability machine; pre-training + RLHF; transformers parallelizelessons/05-llms-explained-briefly.md
6Transformers, the tech behind LLMs (Ch. 5)Tokens → embeddings → attention/MLP stack → unembedding → softmax (with temperature)lessons/06-transformers.md
7Attention in transformers, step-by-step (Ch. 6)softmax(QKᵀ/√d) with causal mask weights value vectors added to embeddings; 96 heads × 96 layerslessons/07-attention.md
8How might LLMs store facts (Ch. 7)MLP = up-projection, ReLU AND-gate, down-projection adding a fact direction; superpositionlessons/08-mlps-store-facts.md
9But what is cross-entropy? (Compression is Intelligence, Pt. 2)Cross-entropy = cost of using the wrong code; it is the only loss minimized at model = datalessons/09-cross-entropy.md
10But how do AI images and videos actually work? (guest: Welch Labs)CLIP + diffusion as reversed Brownian motion; noise during sampling, DDIM, classifier-free guidancelessons/10-diffusion-models.md

Cross-lesson rules, formulas and techniques: cheatsheet.md.

How to use this skill

  • Concept question → answer from the mental models above, then open the one lesson file that covers it for detail and the video link.
  • "How many parameters / what dimension" → GPT-3 section above, then L6–L8.
  • Building an explanation for a learner → follow the course order; the course deliberately teaches the plain MNIST network first because every later piece (MLP blocks, backprop at scale) reuses it.
  • Caveat: the MNIST net, GPT-3 figures and examples are the course's teaching choices (the MNIST net is, in the course's words, 80s/90s-era technology; GPT-3 is used because its numbers are public); the course itself notes modern models are bigger and their numbers mostly unpublished.
reverse-engineer-anythingmorluto/reaReverse engineer native, managed, Electron/JavaScript, packaged, firmware, and browser targets with REA, and build agent environments (AgentEnv bundles) of web applications from a live instance or a capture, and of Linux desktop apps with a task from a request. Use shipped-artifact or requested runtime evidence to explain features, compare versions, decompile code, or guide a reconstruction. Skip REA for ordinary source-repository architecture analysis.Stars: 471kUpdated Claude CodeCodexVerifiedInstall
diagnosing-superpowersobra/superpowersUse when a superpowers session went wrong and your human partner wants to know why — repeated work, ignored plans, stumbles, poor results, a skill that didn't fire, "it took too long", "why is it so expensive", "what is it doing" — or wants to build a bug report for the superpowers maintainers, for the current session or a past one identified by id or path, on any harness.Stars: 322kUpdated Claude CodeCodexVerifiedInstall
dispatching-parallel-agentsobra/superpowersUse when facing 2+ independent tasks that can be worked on without shared state or sequential dependenciesStars: 322kUpdated Claude CodeCodexVerifiedInstall
executing-plansobra/superpowersUse when executing an implementation plan in the current session as the implementer yourself — your human partner chose inline execution, or no subagent tool is availableStars: 322kUpdated Claude CodeCodexVerifiedInstall