Skip to content

michalstrnadel/source-to-skill / examples/neural-networks-3blue1brown

neural-networks-3blue1brown

Verified

Course skill distilled from 3Blue1Brown's "Neural networks" YouTube playlist (10 videos) - from a plain MNIST digit classifier (neurons, weights, biases, sigmoid/ReLU), through cost functions, gradient descent, stochastic gradient descent and backpropagation with its chain-rule calculus, to large language models and transformers (tokens, embeddings, softmax and temperature, attention heads with query/key/value, masking, MLP blocks storing facts, superposition, the full GPT-3 175B parameter tally), cross-entropy loss, KL divergence and distillation, and diffusion image/video models (CLIP, DDPM, DDIM, classifier-free guidance). Load when explaining or teaching how neural networks, backprop, transformers, attention, LLM training losses or diffusion models work, when someone needs an intuition-first mental model or a concrete number (e.g. GPT-3 dimensions), or asks "what does this 3Blue1Brown lesson say about X".

lessons/09-cross-entropy.md

View on GitHub
https://www.youtube.com/watch?v=GlYgs6v2YfU

# Lesson 9 — But what is cross-entropy? (Compression is Intelligence, Part 2)

Part 2 of a separate series; Part 1 (entropy, optimal codes) is not in this
playlist, but the needed results are restated. Thesis: cross-entropy arises
naturally in compression, is also the LLM training loss, and that shared
formula hints that training a language model is training a compressor.

## Hook: *Language Trees and Zipping* (2002) ([1:03](https://www.youtube.com/watch?v=GlYgs6v2YfU&t=63s))

- Cluster documents by language and even recover the tree of language lineage
  using only **gzip**, no linguistics.
- Trick: append a snippet of B to A, compress, compare with compressing A
  alone. Small difference ⇒ B's patterns resemble A's. A distance built on
  this recovered the lineage tree; the same trick works for authorship.
- It is only an empirical, rough estimate of cross-entropy (documents aren't
  distributions; gzip just replaces repeats with pointers and is far from the
  Shannon limit) — but useful. ([12:25](https://www.youtube.com/watch?v=GlYgs6v2YfU&t=745s))

## Information and entropy (recap) ([3:08](https://www.youtube.com/watch?v=GlYgs6v2YfU&t=188s))

- Robot instructions up/down/left/right with probabilities 1/2, 1/4, 1/8, 1/8:
  optimal code uses 1, 2, 3, 3 bits.
- General rule: an optimal code spends **−log₂ p** bits on a symbol (Shannon's
  **information content**; "how many times do you halve to get there"). ([4:09](https://www.youtube.com/watch?v=GlYgs6v2YfU&t=249s))
- A message's information = sum over its symbols; optimal encoding length ≈
  that total, so fractional bits are meaningful. ([5:11](https://www.youtube.com/watch?v=GlYgs6v2YfU&t=311s))
- **Entropy** H(Q) = Σ qᵢ · (−log₂ qᵢ): the average bits per symbol under the
  optimal code for Q. Visualize as bars with width qᵢ, height −log₂ qᵢ.

## Cross-entropy

- New reality P: up 1/8, down 1/8, left 1/4, right 1/2, but the old code is
  hard-wired. Average cost = **2.625 bits/symbol** — the cross-entropy. ([5:11](https://www.youtube.com/watch?v=GlYgs6v2YfU&t=311s))
- Definition: **H(P, Q) = Σ pᵢ · (−log₂ qᵢ)** — average bits when reality is P
  but the code is optimized for Q. Bars: width pᵢ, height −log₂ qᵢ. Notation
  conventions in the wild vary; think about the sum itself. ([7:15](https://www.youtube.com/watch?v=GlYgs6v2YfU&t=435s))
- Order matters: ([9:20](https://www.youtube.com/watch?v=GlYgs6v2YfU&t=560s))
  - Q = 50/50, P = 90/10 → H(Q) = 1 bit, cross-entropy = 1 bit (every symbol
    costs 1 bit, so weights don't matter).
  - Q = 90/10, P = 50/50 → H(Q) < 1 bit, but cross-entropy ≈ **1.74 bits**.
- **Key property:** fix P, vary Q → cross-entropy is minimized exactly when
  **Q = P**, and that minimum equals **H(P)**. (Traced over all P, the minima
  draw the entropy curve.) ([10:22](https://www.youtube.com/watch?v=GlYgs6v2YfU&t=622s))
- General use: quantify how different the patterns of one setting are from
  another's.

## Pre-training loss ([15:32](https://www.youtube.com/watch?v=GlYgs6v2YfU&t=932s))

- An LM maps any token sequence to a distribution over the next token. Train
  with a loss; once the loss is defined, gradient descent and backprop "take it
  from there" — the engineer's key design question is the loss.
- Loss = **average information per token from the model's perspective**: for
  every prefix in the data, take −log(probability the model gave the true next
  token), average over all tokens in the training set. Models produce all these
  probabilities in one pass. ([16:34](https://www.youtube.com/watch?v=GlYgs6v2YfU&t=994s))
- A smart model is rarely surprised (low loss); a confused one is surprised
  constantly; −log punishes very low probabilities steeply.
- ML uses the **natural log**: differs from log₂ by a constant absorbed into
  the learning rate, and derivatives are cleaner. ([18:38](https://www.youtube.com/watch?v=GlYgs6v2YfU&t=1118s))
- Minimizing it ideally approaches the **entropy of language**, leaving a
  powerful general predictor. Glossed over: batching, optimizers, engineering at
  scale.

## Why it's called cross-entropy (and why log is forced)

- The common explanation — cross-entropy against a one-hot distribution on the
  true token, which collapses to −log q — is unsatisfying: it doesn't explain
  *why*. ([20:42](https://www.youtube.com/watch?v=GlYgs6v2YfU&t=1242s))
- Better: take a context like "My name is ___" that appears many times. Let Q
  be the model's distribution and P the frequency of each name in the data.
  The average loss over all instances = Σ pᵢ · F(qᵢ) for some decreasing loss
  function F. ([21:44](https://www.youtube.com/watch?v=GlYgs6v2YfU&t=1304s))
- With F = −log, this **is** the cross-entropy of the model relative to the
  data, minimized exactly when model = data statistics.
- Conversely, if you demand the average loss be minimized **only** when the
  model matches the data, a Lagrange-multiplier argument (minimize subject to
  Σqᵢ = 1) forces F′(q) ∝ 1/q — **only logarithms** qualify. "Your hand is
  forced." ([24:52](https://www.youtube.com/watch?v=GlYgs6v2YfU&t=1492s))

## Distillation ([25:53](https://www.youtube.com/watch?v=GlYgs6v2YfU&t=1553s))

- Train a small model to mimic a big one: at each token, loss = cross-entropy
  of the small model's full distribution relative to the big model's full
  distribution, rather than against the single true next token.
- Much richer signal: like learning chess from a stronger player weighing all
  good moves versus watching one game. For "My name is ___", one example
  already carries a whole reasonable name distribution.

## KL divergence

- **KL(P‖Q) = cross-entropy − entropy of P**: bits per symbol wasted by using a
  poorly optimized code. (Robot example, derived from the numbers above:
  2.625 − 1.75 = 0.875 bits.) ([31:02](https://www.youtube.com/watch?v=GlYgs6v2YfU&t=1862s))
- Acts as a distance between distributions: zero when equal, grows as they
  differ, but **asymmetric**.
- Exercises posed: show the compact KL formula equals the difference form;
  interpret the KL bar diagram; what would happen if distillation used KL
  divergence instead of cross-entropy? ([33:05](https://www.youtube.com/watch?v=GlYgs6v2YfU&t=1985s))

## Coming next (not in playlist)

Turning a predictor into a compressor (bits ≈ the text's information content
from the model's perspective), which makes cross-entropy training equivalent
to training the best text compressor — the basis for "compression is
intelligence".
reverse-engineer-anythingmorluto/reaReverse engineer native, managed, Electron/JavaScript, packaged, firmware, and browser targets with REA, and build agent environments (AgentEnv bundles) of web applications from a live instance or a capture, and of Linux desktop apps with a task from a request. Use shipped-artifact or requested runtime evidence to explain features, compare versions, decompile code, or guide a reconstruction. Skip REA for ordinary source-repository architecture analysis.Stars: 472kUpdated Claude CodeCodexVerifiedInstall
diagnosing-superpowersobra/superpowersUse when a superpowers session went wrong and your human partner wants to know why — repeated work, ignored plans, stumbles, poor results, a skill that didn't fire, "it took too long", "why is it so expensive", "what is it doing" — or wants to build a bug report for the superpowers maintainers, for the current session or a past one identified by id or path, on any harness.Stars: 323kUpdated Claude CodeCodexVerifiedInstall
dispatching-parallel-agentsobra/superpowersUse when facing 2+ independent tasks that can be worked on without shared state or sequential dependenciesStars: 323kUpdated Claude CodeCodexVerifiedInstall
executing-plansobra/superpowersUse when executing an implementation plan in the current session as the implementer yourself — your human partner chose inline execution, or no subagent tool is availableStars: 323kUpdated Claude CodeCodexVerifiedInstall