Skip to content

michalstrnadel/source-to-skill / examples/neural-networks-3blue1brown

neural-networks-3blue1brown

Verified

Course skill distilled from 3Blue1Brown's "Neural networks" YouTube playlist (10 videos) - from a plain MNIST digit classifier (neurons, weights, biases, sigmoid/ReLU), through cost functions, gradient descent, stochastic gradient descent and backpropagation with its chain-rule calculus, to large language models and transformers (tokens, embeddings, softmax and temperature, attention heads with query/key/value, masking, MLP blocks storing facts, superposition, the full GPT-3 175B parameter tally), cross-entropy loss, KL divergence and distillation, and diffusion image/video models (CLIP, DDPM, DDIM, classifier-free guidance). Load when explaining or teaching how neural networks, backprop, transformers, attention, LLM training losses or diffusion models work, when someone needs an intuition-first mental model or a concrete number (e.g. GPT-3 dimensions), or asks "what does this 3Blue1Brown lesson say about X".

lessons/01-what-is-a-neural-network.md

View on GitHub
https://www.youtube.com/watch?v=aircAruvnKk

# Lesson 1 — But what is a neural network? (Deep Learning Chapter 1)

Goal: understand the *structure* of a plain "vanilla" network (no frills)
that recognizes handwritten digits; learning comes in lesson 2. The plain
form is presented as the prerequisite for every modern variant.

## The task

- Input: a 28×28 grayscale image of a digit. Output: which digit (0–9).
- Writing this as an explicit program is "dauntingly difficult" even though
  brains do it effortlessly — the motivation for learning from data.

## Anatomy

- **Neuron** = a thing that holds a number between 0 and 1 (its
  **activation**). Later refined: each neuron is really a *function* of the
  previous layer's outputs.
- **Input layer**: 784 neurons (28 × 28), activation = pixel brightness
  (0 black → 1 white). ([3:08](https://www.youtube.com/watch?v=aircAruvnKk&t=188s))
- **Output layer**: 10 neurons, one per digit; the brightest is the network's
  answer.
- **Hidden layers**: two layers of 16 neurons each — an admittedly arbitrary
  choice (16 "fit nicely on the screen"); structure is open to experiment. ([4:09](https://www.youtube.com/watch?v=aircAruvnKk&t=249s))
- Activations in one layer determine the next, loosely analogous to
  biological neurons firing others.

## Why layers might work (the hope) ([5:11](https://www.youtube.com/watch?v=aircAruvnKk&t=311s))

- Digits decompose into parts: 9 = upper loop + right line; 8 = two loops;
  4 ≈ three lines.
- Hope: second-to-last layer detects such sub-components; the layer before
  detects little edges; the last layer combines components into digits.
- Same layered-abstraction idea applies elsewhere, e.g. speech: raw audio →
  sounds → syllables → words → phrases → thoughts. ([8:16](https://www.youtube.com/watch?v=aircAruvnKk&t=496s))
- (Lesson 2 shows the trained network does *not* actually do this.)

## Weights, bias, nonlinearity

- Each connection gets a **weight** (just a number). A neuron computes the
  weighted sum of all previous activations. ([9:18](https://www.youtube.com/watch?v=aircAruvnKk&t=558s))
- Visualize a neuron's 784 weights as a pixel grid: green = positive, red =
  negative. To detect an edge in a region: positive weights in the region,
  negative weights around it — the sum is largest when the middle is bright and
  the surroundings dark.
- Squash the sum into (0, 1) with the **sigmoid** (logistic curve): very
  negative → ≈0, very positive → ≈1, steepest around 0. ([10:18](https://www.youtube.com/watch?v=aircAruvnKk&t=618s))
- **Bias**: a number added before squashing (e.g. −10 if the neuron should
  only fire when the sum exceeds 10). Weights = *which pattern*; bias = *how
  strong before it activates*. ([11:20](https://www.youtube.com/watch?v=aircAruvnKk&t=680s))
- Counting: 784 × 16 weights + 16 biases for the first hidden layer alone;
  ~13,000 weights and biases total — the knobs that "learning" sets. ([12:23](https://www.youtube.com/watch?v=aircAruvnKk&t=743s))

## Compact notation ([13:26](https://www.youtube.com/watch?v=aircAruvnKk&t=806s))

- Activations of a layer → column vector `a`. Weights → matrix `W` (row j =
  all connections into neuron j of the next layer). Biases → vector `b`.
- Layer transition: `a⁽¹⁾ = σ(W a⁽⁰⁾ + b)`, sigmoid applied element-wise.
- Benefits: simpler code and faster, because libraries heavily optimize
  matrix multiplication. "So much of machine learning" is linear algebra.

## The whole network is a function ([15:31](https://www.youtube.com/watch?v=aircAruvnKk&t=931s))

784 numbers in → 10 numbers out, with ~13,000 parameters, iterated
matrix-vector products and sigmoids. Its complexity is reassuring: anything
simpler would have little hope of recognizing digits.

Why bother understanding individual weights: when a network misbehaves (or
works for unexpected reasons), knowing what weights and biases mean gives you
a starting point to change the structure and to challenge assumptions.

## Coda: sigmoid vs. ReLU (with Lisha Li) ([16:33](https://www.youtube.com/watch?v=aircAruvnKk&t=993s))

- Sigmoid was used by early networks, motivated by the active/inactive neuron
  analogy; relatively few modern networks use it — "kind of old school".
- **ReLU** (rectified linear unit) = `max(0, a)`: identity above the
  threshold, zero below. Sigmoid made deep networks hard to train; ReLU
  "happened to work very well" for very deep networks. ([17:34](https://www.youtube.com/watch?v=aircAruvnKk&t=1054s))
reverse-engineer-anythingmorluto/reaReverse engineer native, managed, Electron/JavaScript, packaged, firmware, and browser targets with REA, and build agent environments (AgentEnv bundles) of web applications from a live instance or a capture, and of Linux desktop apps with a task from a request. Use shipped-artifact or requested runtime evidence to explain features, compare versions, decompile code, or guide a reconstruction. Skip REA for ordinary source-repository architecture analysis.Stars: 472kUpdated Claude CodeCodexVerifiedInstall
diagnosing-superpowersobra/superpowersUse when a superpowers session went wrong and your human partner wants to know why — repeated work, ignored plans, stumbles, poor results, a skill that didn't fire, "it took too long", "why is it so expensive", "what is it doing" — or wants to build a bug report for the superpowers maintainers, for the current session or a past one identified by id or path, on any harness.Stars: 323kUpdated Claude CodeCodexVerifiedInstall
dispatching-parallel-agentsobra/superpowersUse when facing 2+ independent tasks that can be worked on without shared state or sequential dependenciesStars: 323kUpdated Claude CodeCodexVerifiedInstall
executing-plansobra/superpowersUse when executing an implementation plan in the current session as the implementer yourself — your human partner chose inline execution, or no subagent tool is availableStars: 323kUpdated Claude CodeCodexVerifiedInstall