michalstrnadel/source-to-skill / examples/neural-networks-3blue1brown
neural-networks-3blue1brown
VerifiedCourse skill distilled from 3Blue1Brown's "Neural networks" YouTube playlist (10 videos) - from a plain MNIST digit classifier (neurons, weights, biases, sigmoid/ReLU), through cost functions, gradient descent, stochastic gradient descent and backpropagation with its chain-rule calculus, to large language models and transformers (tokens, embeddings, softmax and temperature, attention heads with query/key/value, masking, MLP blocks storing facts, superposition, the full GPT-3 175B parameter tally), cross-entropy loss, KL divergence and distillation, and diffusion image/video models (CLIP, DDPM, DDIM, classifier-free guidance). Load when explaining or teaching how neural networks, backprop, transformers, attention, LLM training losses or diffusion models work, when someone needs an intuition-first mental model or a concrete number (e.g. GPT-3 dimensions), or asks "what does this 3Blue1Brown lesson say about X".
| File | Kind | Size |
|---|---|---|
| SKILL.md | skill md | 9.3 KB |
| cheatsheet.md | markdown | 5.8 KB |
| evals/evals.json | config | 3.8 KB |
| evals/last-run.json | config | 2.1 KB |
| lessons/01-what-is-a-neural-network.md | markdown | 4.5 KB |
| lessons/02-gradient-descent.md | markdown | 4.9 KB |
| lessons/03-backpropagation-intuition.md | markdown | 3.7 KB |
| lessons/04-backpropagation-calculus.md | markdown | 2.9 KB |
| lessons/05-llms-explained-briefly.md | markdown | 3.1 KB |
| lessons/06-transformers.md | markdown | 6.8 KB |
| lessons/07-attention.md | markdown | 5.8 KB |
| lessons/08-mlps-store-facts.md | markdown | 5.2 KB |
| lessons/09-cross-entropy.md | markdown | 6.5 KB |
| lessons/10-diffusion-models.md | markdown | 7.4 KB |
| source.json | config | 1.8 KB |
lessons/05-llms-explained-briefly.md
View on GitHubhttps://www.youtube.com/watch?v=LPZh9BOjkQs
# Lesson 5 — Large Language Models explained briefly
A short, non-technical overview that bridges the MNIST lessons and the
transformer chapters.
## What an LLM is
- A sophisticated mathematical function that predicts the next word for any
text — as a **probability for every possible next word**, not one certain
answer.
- **Chatbot recipe:** write text describing an interaction between a user and
a hypothetical AI assistant, append the user's input, then repeatedly
predict what the assistant would say next.
- Sampling less likely words at random makes output look more natural; so the
deterministic model gives different answers to the same prompt. ([1:02](https://www.youtube.com/watch?v=LPZh9BOjkQs&t=62s))
## Training
- Behavior is fully determined by many continuous **parameters** (weights);
"large" = hundreds of billions of them. Nobody sets them by hand — they
start random (gibberish output) and are refined on example text. ([2:04](https://www.youtube.com/watch?v=LPZh9BOjkQs&t=124s))
- One training step: feed all but the last word of an example, compare the
prediction to the true last word, and use **backpropagation** to make the
true word slightly more likely and all others slightly less likely.
- Repeated over many trillions of examples, predictions improve on training
data *and* on unseen text. ([3:07](https://www.youtube.com/watch?v=LPZh9BOjkQs&t=187s))
- Scale: reading GPT-3's training text non-stop would take a human **over
2,600 years**. At one billion additions/multiplications per second, the
operations to train the largest models would take **well over 100 million
years**.
- This is **pre-training**. Autocompleting internet text ≠ being a good
assistant, so chatbots also get **reinforcement learning with human
feedback** (RLHF): workers flag unhelpful or problematic predictions and the
corrections further tune the parameters. ([3:07](https://www.youtube.com/watch?v=LPZh9BOjkQs&t=187s))
## Why transformers
- Training at this scale relies on **GPUs**, chips that run many operations
in parallel — but not all models parallelize well. ([4:11](https://www.youtube.com/watch?v=LPZh9BOjkQs&t=251s))
- Before 2017 most language models processed text one word at a time. A
Google team introduced the **transformer**, which takes in all the text at
once, in parallel.
- Steps:
1. Each word → a long list of numbers (training only works with continuous
values), which may encode the word's meaning.
2. **Attention** lets those lists "talk to one another" and refine meaning by
context, in parallel (e.g. "bank" → riverbank). ([5:13](https://www.youtube.com/watch?v=LPZh9BOjkQs&t=313s))
3. **Feed-forward networks** add capacity to store patterns learned in
training.
4. Many alternating iterations of both; finally a function on the **last**
vector produces next-word probabilities.
- Researchers design the framework; the specific behavior **emerges** from
the tuned parameters, which makes it very hard to explain why a model makes
a given prediction. ([6:16](https://www.youtube.com/watch?v=LPZh9BOjkQs&t=376s))
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| reverse-engineer-anythingmorluto/reaReverse engineer native, managed, Electron/JavaScript, packaged, firmware, and browser targets with REA, and build agent environments (AgentEnv bundles) of web applications from a live instance or a capture, and of Linux desktop apps with a task from a request. Use shipped-artifact or requested runtime evidence to explain features, compare versions, decompile code, or guide a reconstruction. Skip REA for ordinary source-repository architecture analysis. | Stars: 472k | Updated | Claude CodeCodex | Verified | Install |
| diagnosing-superpowersobra/superpowersUse when a superpowers session went wrong and your human partner wants to know why — repeated work, ignored plans, stumbles, poor results, a skill that didn't fire, "it took too long", "why is it so expensive", "what is it doing" — or wants to build a bug report for the superpowers maintainers, for the current session or a past one identified by id or path, on any harness. | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |
| dispatching-parallel-agentsobra/superpowersUse when facing 2+ independent tasks that can be worked on without shared state or sequential dependencies | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |
| executing-plansobra/superpowersUse when executing an implementation plan in the current session as the implementer yourself — your human partner chose inline execution, or no subagent tool is available | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |