🐇 Down the Rabbit Hole

How Large Language Models Actually Work — A Guided Tour

📺 Video Lesson • 51 min • Module 1

🎬 The Video

"I Built an LLM From Scratch" — CJ Reynolds (Syntax)

I Built an LLM From Scratch - YouTube thumbnail
▶
51:47
📺 Watch on YouTube
⏱️ 51 min 48 sec 📂 Companion app: w3cj/how-llms-work 🏷️ #machinelearning #typescript #webdeveloper

🎯 Learning Objectives

By the end of this lesson, you will be able to:

🧩 Distinguish pattern-matching "AI" from actual neural networks
📊 Explain backpropagation and why depth matters
🔤 Describe how tokenization turns text into numbers
🌐 Understand what word embeddings capture semantically
🏗️ Identify the key components of a transformer (attention, layers, normalization)
💡 Appreciate that LLMs are built from simple, learnable operations — not magic
🐛 Your progress 0 / 5 sections

📖 Guided Viewing

Watch each section, then check it off. Click a header to expand notes and checkpoints.

✓ 1 Simple Chat — Pattern Matching ▾

⏱ ~5 min

CJ builds an ELIZA-style chatbot using if-statements and pattern matching. It streams responses word-by-word via Server-Sent Events — the same streaming plumbing real LLMs use, but with zero intelligence behind it.

🔍 Watch for: How does the "fake AI" compare to a real LLM? Notice that the streaming infrastructure is identical — the illusion of intelligence starts long before the intelligence itself.

💬 Key insight: "Same plumbing as ChatGPT, zero intelligence." This sets the baseline — without neural networks, there's no learning, only hardcoded rules.

✓ 2 XOR Neural Net — Backpropagation ▾

⏱ ~10 min

A single-layer perceptron fails on XOR (proving Minsky & Papert's 1969 result). A multi-layer network succeeds via backpropagation — the same algorithm every neural network uses today, from image classifiers to LLMs.

🔍 Watch for: The live training visualization. Watch the decision boundary shift as the network learns. Notice why a single layer can't solve XOR — this is why depth matters.

💬 Key insight: Backpropagation is how every neural network learns. The math is just the chain rule from calculus — propagating errors backward through the layers so each neuron can adjust.

✓ 3 Basic Tokenizer — BPE From Scratch ▾

⏱ ~10 min

Byte Pair Encoding (BPE) from scratch. CJ trains a tokenizer on whatever text you input — watch merge steps animate as the algorithm builds a vocabulary from individual characters up to common words and subwords.

🔍 Watch for: The animated merge visualization. See how "low" + "er" → "lower" as the algorithm learns common subword patterns. This is how your typed text becomes token IDs.

💬 Key insight: Every word gets broken into subword tokens. "ChatGPT" might become ["Chat", "G", "PT"]. Tokenization is the bridge between human language and neural network math.

✓ 4 Train Embeddings — Word2Vec Skip-Gram ▾

⏱ ~10 min

Trains word embeddings from scratch using Word2Vec skip-gram with negative sampling. Watch vectors learn that words used in similar contexts cluster together in vector space.

🔍 Watch for: The vector space visualization — words moving into clusters by meaning. "King" - "Man" + "Woman" ≈ "Queen" is the classic result that shows embeddings capture relationships, not just definitions.

💬 Key insight: "You shall know a word by the company it keeps." Words that appear in similar contexts get similar vector representations. This is how LLMs build understanding of meaning without being explicitly taught definitions.

✓ 5 Train Transformer — GPT From Scratch ▾

⏱ ~15 min

A decoder-only transformer built entirely from scratch — no ML libraries. Every operation hand-implemented: multi-head causal self-attention, layer normalization, feed-forward layers, backpropagation, and Adam optimization. Uses multi-threaded data parallelism via SharedArrayBuffer for training speed.

🔍 Watch for: The attention mechanism visualization — see how each token "looks at" every previous token to decide which context matters. The multi-head attention lets the model attend to different kinds of relationships simultaneously.

💬 Key insight: The transformer is just matrix multiplications + softmax + layer norm. There's no single "intelligence switch" — the magic is emergent from thousands of simple operations at massive scale.

📚 References from the codebase: Vaswani et al. "Attention Is All You Need" (2017), Radford et al. "GPT-1" (2018), Holtzman et al. "The Curious Case of Neural Text Degeneration" (2019).

📚 Key Vocabulary 30 terms

Tap or click a term to reveal its plain-language definition. Terms are grouped by course module.

🐇 Module 1 — How LLMs Work

LLM (Large Language Model)
A type of AI trained on massive amounts of text to predict and generate human-like language. GPT, Claude, Gemini — they're all LLMs.
🔤 tap to reveal
Tokens
The basic units an LLM reads and writes. A word like "hello" might be one token; a longer or unusual word might be split into several. Think of them as the model's alphabet.
🔤 tap to reveal
Tokenization
The process of breaking text into tokens. BPE (Byte Pair Encoding) is the most common algorithm — it starts with individual characters and merges frequent pairs into larger tokens.
🔤 tap to reveal
Context Window
The maximum amount of text (in tokens) an LLM can "see" at once when generating a response. Think of it as the model's working memory — beyond this, it forgets.
🔤 tap to reveal
Embeddings
Dense vectors (lists of numbers) that represent the meaning of words or tokens. Words used in similar contexts end up with similar vectors — "king" and "queen" are closer than "king" and "toaster".
🔤 tap to reveal
Transformer
The neural network architecture (Vaswani et al., 2017) that powers all modern LLMs. Built from stacked attention and feed-forward layers — the "T" in GPT and "T" in BERT.
🔤 tap to reveal
Attention (Self-Attention)
The mechanism that lets each token "look at" every other token in the context to decide which parts matter most. When the model reads "the cat sat on the ___", attention helps it focus on "cat" and "sat" to predict "mat".
🔤 tap to reveal
Backpropagation
The algorithm that trains neural networks by computing errors backward from the output through every layer, adjusting each connection to reduce mistakes next time. Same algorithm from XOR nets to GPT-5.
🔤 tap to reveal
Inference
When a trained model is actually used — you give it input, it runs a forward pass through the network, and produces output. Training is the practice; inference is the performance.
🔤 tap to reveal
Logit / Softmax
Logits are the raw scores the model assigns to every possible next token before picking one. Softmax turns those scores into probabilities (0–100%) that sum to 1, so the model can make a weighted choice.
🔤 tap to reveal

🎩 Module 2 — Prompting & Interaction

Prompt Engineering
The craft of designing input text to get the best possible output from an LLM. Like asking a good question — specific, structured, and with the right context — rather than a vague one.
🔤 tap to reveal
System Prompt
An invisible instruction that sets the model's behaviour, role, and constraints before the conversation starts. Like giving an actor their character notes before they step on stage.
🔤 tap to reveal
Chain-of-Thought (CoT)
A prompting technique that asks the model to show its reasoning step by step before answering. Like a maths teacher saying "show your working" — it dramatically improves accuracy on complex problems.
🔤 tap to reveal
Zero-shot Learning
Asking an LLM to perform a task with no examples — just a description. "Translate this to French:" without showing any translations first. Surprisingly effective with modern models.
🔤 tap to reveal
Few-shot Learning
Giving the model 2–5 examples of what you want before asking it to do the task. "Here are three haikus. Now write one about a cat." — the examples teach the pattern.
🔤 tap to reveal
Temperature
A dial that controls randomness. Low temperature (0.1) = the model picks the most likely word every time (factual, repeatable). High temperature (1.5) = more surprises (creative, varied).
🔤 tap to reveal
Top-p / Top-k Sampling
Two knobs that narrow which tokens the model can choose from. Top-k only considers the k most likely next words; Top-p considers tokens until their combined probability reaches p%. Both reduce weird outputs.
🔤 tap to reveal
Hallucinations
When an LLM confidently states something that isn't true. The model isn't lying — it's just predicting the most plausible-sounding next word. A bit like the Mad Hatter: charming, but don't trust his directions.
🔤 tap to reveal

🫖 Module 3 — Customising & Enhancing

Fine-tuning
Taking a pre-trained LLM and training it further on a smaller, specific dataset. Like sending a general doctor to specialise in cardiology — the base knowledge is there, you're sharpening one skill.
🔤 tap to reveal
PEFT (Parameter-Efficient Fine-Tuning)
A family of techniques that fine-tune a model by adjusting only a tiny fraction of its parameters, keeping most of the original model frozen. Much cheaper and faster than full fine-tuning.
🔤 tap to reveal
LoRA (Low-Rank Adaptation)
The most popular PEFT method. Instead of retraining the giant model, it trains small "adapter" matrices that plug into the existing layers. Imagine adding training wheels rather than rebuilding the bike.
🔤 tap to reveal
RAG (Retrieval-Augmented Generation)
A technique that lets an LLM look up information from a knowledge base before answering. Instead of guessing, the model searches for relevant documents first — like a student allowed to open their textbook during the exam.
🔤 tap to reveal
Alignment
Training an LLM to behave in ways that are helpful, honest, and harmless. A raw model can predict toxic text just as well as kind text — alignment steers it toward the good stuff.
🔤 tap to reveal
RLHF (Reinforcement Learning from Human Feedback)
The main alignment technique: humans rate model outputs, and those ratings train a reward model that fine-tunes the LLM. Like teaching a dog by saying "good boy!" — the model learns what humans prefer.
🔤 tap to reveal
Safety / Guardrails
Rules and filters that prevent an LLM from generating harmful, biased, or unsafe content. Think of them as the Queen of Hearts' guards — they stop the model from yelling "Off with their heads!" at the wrong moment.
🔤 tap to reveal
Quantization
A compression technique that shrinks an LLM by using fewer bits to store each number (e.g. 16-bit → 4-bit). Makes models smaller and faster to run, at a small cost to quality. How you get a 70B model running on a laptop.
🔤 tap to reveal

♠️ Module 4 — Ecosystem & Future

Agent
An LLM that can use tools, make decisions, and take actions on its own — searching the web, writing files, controlling apps. Like giving Alice a map and letting her explore Wonderland instead of just describing it.
🔤 tap to reveal
Multimodal
An LLM that can understand and generate multiple types of data — text, images, audio, video. GPT-4 with vision and DALL·E built in is multimodal; text-only ChatGPT is not.
🔤 tap to reveal
API (Application Programming Interface)
The way developers send requests to an LLM from their own code. Instead of typing in a chat box, software calls the API — which is how apps like Grammarly or Notion AI get their smarts.
🔤 tap to reveal
Open Source LLMs
Models whose weights are publicly released — anyone can download, inspect, modify, and run them locally (e.g. Llama, Mistral, DeepSeek). Like a recipe book you can change.
🔤 tap to reveal
Proprietary LLMs
Models owned by companies that keep the internals secret — you access them only through an API or app (e.g. GPT-4, Claude, Gemini). Like a restaurant meal: delicious, but you can't see the recipe.
🔤 tap to reveal
Latency
The time it takes for an LLM to start responding after you hit Enter. Lower latency = feels snappy and conversational. Higher latency = you wait, wondering if something broke.
🔤 tap to reveal
Throughput
How many tokens an LLM can generate per second. High throughput means the model can serve many users at once or generate long documents quickly.
🔤 tap to reveal
AGI (Artificial General Intelligence)
A hypothetical future AI that can learn and perform any intellectual task a human can — not just language, but reasoning, planning, creativity, and adapting to novel situations. The white rabbit we're all chasing.
🔤 tap to reveal

📖 View full course glossary →

🧪 Quick Comprehension Check

Test your understanding after watching. No grades — just for you.

1. What makes the ELIZA-style chatbot in Section 1 different from a real LLM?

2. Why does a single-layer perceptron fail on XOR?

3. What are word embeddings?

4. Which paper introduced the Transformer architecture?

💬 Discussion Prompts

Pick one to reflect on, then bring your thoughts to the group.

🫖 The "Magic" Question

After seeing the ELIZA chatbot (if-statements pretending to be intelligent), do you think real LLMs are also "just pattern matching" at a larger scale, or is there a qualitative difference? Where's the line between mimicry and understanding?

🌳 From Acorns to Oak Trees

The XOR neural net uses the same algorithm (backpropagation) as GPT-5. What does it mean that intelligence at every scale — from classifying two inputs to writing poetry — relies on the same simple learning rule? Does that democratize AI or make it more mysterious?

🔤 Alice's Token Tea Party

CJ's tokenizer demonstrates that "ChatGPT" gets broken into subword pieces. How does tokenization affect what an LLM can and cannot understand? Why might a model struggle with unusual names, invented words, or code?

🎩 The Mad Hatter's Architecture

The transformer is built from simple parts: attention, feed-forward layers, normalization. If these are all well-understood mathematical operations, why can't we predict what a trained model will do? What's the difference between understanding the parts and understanding the whole?

📓 Reflection Journal

Write your thoughts. They're saved in your browser (local storage).

✓ Saved

🍵 Next Stop: The Prompt Workshop

Now that you understand how an LLM processes language — tokens → embeddings → attention → prediction — you're ready to learn how to talk to them effectively. Prompting is the craft of giving the model the right context, structure, and constraints for your task.

→ Continue to Lesson 1.3: The Art of the Prompt
🐰 Wonderland AI Literacy Course • Module 1: Foundations
Built with 🫖 by Alice & the Wonderland Team • Video by Syntax (CJ Reynolds)