michalstrnadel/source-to-skill / examples/neural-networks-3blue1brown
neural-networks-3blue1brown
VerifiedCourse skill distilled from 3Blue1Brown's "Neural networks" YouTube playlist (10 videos) - from a plain MNIST digit classifier (neurons, weights, biases, sigmoid/ReLU), through cost functions, gradient descent, stochastic gradient descent and backpropagation with its chain-rule calculus, to large language models and transformers (tokens, embeddings, softmax and temperature, attention heads with query/key/value, masking, MLP blocks storing facts, superposition, the full GPT-3 175B parameter tally), cross-entropy loss, KL divergence and distillation, and diffusion image/video models (CLIP, DDPM, DDIM, classifier-free guidance). Load when explaining or teaching how neural networks, backprop, transformers, attention, LLM training losses or diffusion models work, when someone needs an intuition-first mental model or a concrete number (e.g. GPT-3 dimensions), or asks "what does this 3Blue1Brown lesson say about X".
| File | Kind | Size |
|---|---|---|
| SKILL.md | skill md | 9.3 KB |
| cheatsheet.md | markdown | 5.8 KB |
| evals/evals.json | config | 3.8 KB |
| evals/last-run.json | config | 2.1 KB |
| lessons/01-what-is-a-neural-network.md | markdown | 4.5 KB |
| lessons/02-gradient-descent.md | markdown | 4.9 KB |
| lessons/03-backpropagation-intuition.md | markdown | 3.7 KB |
| lessons/04-backpropagation-calculus.md | markdown | 2.9 KB |
| lessons/05-llms-explained-briefly.md | markdown | 3.1 KB |
| lessons/06-transformers.md | markdown | 6.8 KB |
| lessons/07-attention.md | markdown | 5.8 KB |
| lessons/08-mlps-store-facts.md | markdown | 5.2 KB |
| lessons/09-cross-entropy.md | markdown | 6.5 KB |
| lessons/10-diffusion-models.md | markdown | 7.4 KB |
| source.json | config | 1.8 KB |
lessons/01-what-is-a-neural-network.md
View on GitHubhttps://www.youtube.com/watch?v=aircAruvnKk
# Lesson 1 — But what is a neural network? (Deep Learning Chapter 1)
Goal: understand the *structure* of a plain "vanilla" network (no frills)
that recognizes handwritten digits; learning comes in lesson 2. The plain
form is presented as the prerequisite for every modern variant.
## The task
- Input: a 28×28 grayscale image of a digit. Output: which digit (0–9).
- Writing this as an explicit program is "dauntingly difficult" even though
brains do it effortlessly — the motivation for learning from data.
## Anatomy
- **Neuron** = a thing that holds a number between 0 and 1 (its
**activation**). Later refined: each neuron is really a *function* of the
previous layer's outputs.
- **Input layer**: 784 neurons (28 × 28), activation = pixel brightness
(0 black → 1 white). ([3:08](https://www.youtube.com/watch?v=aircAruvnKk&t=188s))
- **Output layer**: 10 neurons, one per digit; the brightest is the network's
answer.
- **Hidden layers**: two layers of 16 neurons each — an admittedly arbitrary
choice (16 "fit nicely on the screen"); structure is open to experiment. ([4:09](https://www.youtube.com/watch?v=aircAruvnKk&t=249s))
- Activations in one layer determine the next, loosely analogous to
biological neurons firing others.
## Why layers might work (the hope) ([5:11](https://www.youtube.com/watch?v=aircAruvnKk&t=311s))
- Digits decompose into parts: 9 = upper loop + right line; 8 = two loops;
4 ≈ three lines.
- Hope: second-to-last layer detects such sub-components; the layer before
detects little edges; the last layer combines components into digits.
- Same layered-abstraction idea applies elsewhere, e.g. speech: raw audio →
sounds → syllables → words → phrases → thoughts. ([8:16](https://www.youtube.com/watch?v=aircAruvnKk&t=496s))
- (Lesson 2 shows the trained network does *not* actually do this.)
## Weights, bias, nonlinearity
- Each connection gets a **weight** (just a number). A neuron computes the
weighted sum of all previous activations. ([9:18](https://www.youtube.com/watch?v=aircAruvnKk&t=558s))
- Visualize a neuron's 784 weights as a pixel grid: green = positive, red =
negative. To detect an edge in a region: positive weights in the region,
negative weights around it — the sum is largest when the middle is bright and
the surroundings dark.
- Squash the sum into (0, 1) with the **sigmoid** (logistic curve): very
negative → ≈0, very positive → ≈1, steepest around 0. ([10:18](https://www.youtube.com/watch?v=aircAruvnKk&t=618s))
- **Bias**: a number added before squashing (e.g. −10 if the neuron should
only fire when the sum exceeds 10). Weights = *which pattern*; bias = *how
strong before it activates*. ([11:20](https://www.youtube.com/watch?v=aircAruvnKk&t=680s))
- Counting: 784 × 16 weights + 16 biases for the first hidden layer alone;
~13,000 weights and biases total — the knobs that "learning" sets. ([12:23](https://www.youtube.com/watch?v=aircAruvnKk&t=743s))
## Compact notation ([13:26](https://www.youtube.com/watch?v=aircAruvnKk&t=806s))
- Activations of a layer → column vector `a`. Weights → matrix `W` (row j =
all connections into neuron j of the next layer). Biases → vector `b`.
- Layer transition: `a⁽¹⁾ = σ(W a⁽⁰⁾ + b)`, sigmoid applied element-wise.
- Benefits: simpler code and faster, because libraries heavily optimize
matrix multiplication. "So much of machine learning" is linear algebra.
## The whole network is a function ([15:31](https://www.youtube.com/watch?v=aircAruvnKk&t=931s))
784 numbers in → 10 numbers out, with ~13,000 parameters, iterated
matrix-vector products and sigmoids. Its complexity is reassuring: anything
simpler would have little hope of recognizing digits.
Why bother understanding individual weights: when a network misbehaves (or
works for unexpected reasons), knowing what weights and biases mean gives you
a starting point to change the structure and to challenge assumptions.
## Coda: sigmoid vs. ReLU (with Lisha Li) ([16:33](https://www.youtube.com/watch?v=aircAruvnKk&t=993s))
- Sigmoid was used by early networks, motivated by the active/inactive neuron
analogy; relatively few modern networks use it — "kind of old school".
- **ReLU** (rectified linear unit) = `max(0, a)`: identity above the
threshold, zero below. Sigmoid made deep networks hard to train; ReLU
"happened to work very well" for very deep networks. ([17:34](https://www.youtube.com/watch?v=aircAruvnKk&t=1054s))
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| reverse-engineer-anythingmorluto/reaReverse engineer native, managed, Electron/JavaScript, packaged, firmware, and browser targets with REA, and build agent environments (AgentEnv bundles) of web applications from a live instance or a capture, and of Linux desktop apps with a task from a request. Use shipped-artifact or requested runtime evidence to explain features, compare versions, decompile code, or guide a reconstruction. Skip REA for ordinary source-repository architecture analysis. | Stars: 472k | Updated | Claude CodeCodex | Verified | Install |
| diagnosing-superpowersobra/superpowersUse when a superpowers session went wrong and your human partner wants to know why — repeated work, ignored plans, stumbles, poor results, a skill that didn't fire, "it took too long", "why is it so expensive", "what is it doing" — or wants to build a bug report for the superpowers maintainers, for the current session or a past one identified by id or path, on any harness. | Stars: 323k | Updated | Claude CodeCodex | Verified | Install |
| dispatching-parallel-agentsobra/superpowersUse when facing 2+ independent tasks that can be worked on without shared state or sequential dependencies | Stars: 323k | Updated | Claude CodeCodex | Verified | Install |
| executing-plansobra/superpowersUse when executing an implementation plan in the current session as the implementer yourself — your human partner chose inline execution, or no subagent tool is available | Stars: 323k | Updated | Claude CodeCodex | Verified | Install |