1The lecture map
Five acts, eleven chapters. First we see why scale turns a next-word predictor into a general problem solver, then follow the ChatGPT story and the cheapest way to use such a model. Then we build one: tokenizer, pretraining, instruction tuning and RLHF — and finally compare the stages and ask whether bigger is still better. Click a chapter to watch it.
Chapter by chapter what each chapter covers, and the question it leaves for the next
- IWhy largeScale turns a next-word predictor into a general problem solver
- 1
What's New, a Transformer Recap and Why Scale Matters
The field moves in weeks: new models, film-quality video and an agent that “actually does things”. Back to the Transformer: less built-in structure scales better, loss falls along a power law, and some abilities only switch on at scale.
→ So what exactly is a “large” language model? - 2
What Is an LLM? Compression, AGI and How Large Is Large
One model for nearly everything. Predicting the next word well forces a network to compress the internet — and the facts in it. “Large” means billions of parameters, enormous compute and trillions of tokens (36 trillion for Qwen3).
→ Why is all that size worth it? - 3
Why Larger Models: Emergent Abilities and Generalization
Bigger models know more, learn tasks from a few examples or a plain description, and generalize by recombining what they have seen. Even techniques such as chain-of-thought and RLHF only start to help above a certain size.
→ How did these ingredients come together in practice? - IIThe ChatGPT storyGPT-3 + code + instructions + human feedback
- 4
A Case Study: From GPT-3 to ChatGPT, GPT-4 and o1
ChatGPT was GPT-3 plus code training, instruction tuning and RLHF — a breakthrough in interaction more than in knowledge. Then GPT-4, GPT-4V, Sora and o1, and three trends of the modern LLM: large-scale RL, looped Transformers and on-policy distillation.
→ How can you use such a model yourself? - IIIUse itPrompts and agents, no training needed
- 5
Using LLMs, Level 1: Prompt Engineering and Agents
Use the model as a fixed black box: prompt it in a chat window or through an API, or wrap it in an agent with memory, tools, planning and actions — from LangChain and Dify to Manus in the cloud and OpenClaw on your own machine.
→ Agents are easy to build — but what happens inside the model when you train your own? - IVBuild itTokens → pretraining → SFT → RLHF
- 6
Training from Zero: The Pipeline and Tokenization (BPE)
A model learns like a student: words, textbooks, exercises, a teacher's feedback. Step zero is tokenization: subwords balance vocabulary size and sequence length, BPE learns them by merging frequent pairs, and a tokenizer that ignores your language makes everything slower and costlier.
→ With tokens in hand, how does the model read its textbooks? - 7
Pretraining: Language Modeling and Eight Data Tips
Pretraining is next-token prediction over trillions of tokens, and the secret sauce is the data: sources and mixing ratios, continued pretraining, and eight tips — filter, reformat, deduplicate, mix, order, scale (Chinchilla), mask and synthesize.
→ A base model continues text — how do we make it follow instructions? - 8
Instruction Finetuning (SFT): Hypotheses and Tips
Language modeling is not assisting users. SFT trains on instruction–response pairs; it mostly teaches format (LIMA), sometimes knowledge, and generalizes with diverse tasks (FLAN). Tips: distillation, self-instruct, loss on the output only, complex instructions.
→ The token-level loss treats every mistake alike — can we optimize directly for what humans prefer? - 9
RLHF: Reward Models, InstructGPT and ChatGPT
Comparing answers is easier than writing them. Learn a reward model from human comparisons, then optimize the policy against it with PPO. A 1.3B RLHF model beat a 12.9B supervised one; InstructGPT scaled the recipe and ChatGPT added conversation — but human judges have biases.
→ Where do the rest of us get conversation data — and how do the stages compare? - VReflectCompare the stages; what comes after scaling
- 10
Multi-turn Data, PlatoLM and Pretraining vs. SFT vs. RL
ShareGPT gave open models real conversations; PlatoLM trains a user simulator to ask like people do. Pretraining and SFT share one loss, so documents can be turned into Q&A. RL generalizes where SFT memorizes: incentivize, don't teach.
→ If data and RL matter so much, do we still need ever-larger models? - 11
Small LLMs, the Densing Law and the End of Scaling?
Data and hardware put walls in front of scaling. Small models on phones, Phi-4 built on synthetic data, and capability density that keeps rising. Pretraining as we know it may end — next come test-time scaling, agents, denser models and applications.
2Try it in Colab
Three short notebooks turn the lecture into experiments. Click Open in Colab, then Runtime → Run all. Each ends with exercises. All three run on a free CPU runtime; notebook 2 trains six small models, so a free T4 GPU (Runtime → Change runtime type) cuts it from about ten minutes to a couple. To fine-tune and RL-train real models, continue with Tutorial 2: Training Your Own LLMs.
Tokenizers: BPE from scratch
- Replay slide 83's merges: aaabdaaabac → XdXac
- Train your own BPE and watch “tokenizing” split into subwords
- See the original LLaMA tokenizer turn 28 Chinese characters into 35 tokens
- Compare GPT-2, LLaMA and Qwen on English, Chinese and Arabic
.ipynb · CPUScaling laws: fit your own power law
- Train six tiny GPTs (3K–800K parameters) on Shakespeare
- Fit loss = a·N−α + c on a log-log plot and extrapolate
- Split a compute budget between parameters and tokens with the Chinchilla law
- See two fits of the same data disagree (~70 vs. ~20 tokens per parameter)
.ipynb · CPU (T4 GPU faster)Base vs. instruction-tuned: prompting a small LLM
- Ask SmolLM2-135M base and instruct the moon-landing question of slide 106
- Zero-shot vs. few-shot sentiment, scored by next-token probabilities
- Look at the chat template: what SFT data really looks like
- Build the “loss on the output only” mask of slide 119
.ipynb · CPU3Concept index
Key terms of the lecture. Each link opens the video at the slide where the idea is explained.
4What's new in 2026
The lecture opens with the news of early 2026 (slide 2) and closes Chapter 4 with three trends of the modern LLM (slide 59). Six things to watch:
“State of the art” lasts weeks
Around the 2026 Spring Festival almost every major Chinese lab shipped a new model, ByteDance released a new video model, and OpenAI followed with GPT-5.4 — which is why the course teaches principles, not model names.
Agents that actually do things
OpenClaw runs on your own machine, talks to you through your chat apps, remembers, browses and runs commands — and became popular far beyond developers. The lecture's homework: install it and find where it fails.
RL becomes a main training stage
After o1 and DeepSeek-R1, reinforcement learning is no longer a final polish: it is a major, compute-heavy stage, especially for reasoning and agentic tasks.
Depth by looping, small models by on-policy distillation
Looped Transformers reuse one block several times for more computation per token without more parameters. On-policy distillation lets a teacher grade the student's own answers token by token.
Models build the data for the next models
Qwen3 was pretrained on 36 trillion tokens in 119 languages, with text extracted from PDFs and trillions of synthesized tokens; 40% of Phi-4's pretraining comes from synthetic data.
Denser, not just bigger
With data running out and servers limited in memory, the focus shifts from the scaling law to capability density: the same ability with fewer parameters. The densing-law paper finds that the maximum density of open models has doubled roughly every three months.
5Quotes & reading
"The biggest lesson that can be read from 70 years of AI research is that general methods that leverage computation are ultimately the most effective, and by a large margin."
The idea behind slide 9: less hand-built structure, more scale.
"The dialogue format makes it possible for ChatGPT to answer followup questions, admit its mistakes, challenge incorrect premises, and reject inappropriate requests."
ChatGPT's breakthrough was in interaction and alignment, not in raw knowledge.
"Almost all knowledge in large language models is learned during pretraining, and only limited instruction tuning data is necessary to teach models to produce high quality output."
The superficial alignment hypothesis of slide 112 — with its question mark.
"Pre-training as we know it will end. … Data is not growing: we have but one internet, the fossil fuel of AI."
Why the lecture ends with denser models, reasoning and agents instead of “just scale”.
Further reading
- Andrej Karpathy: Intro to Large Language Models (1-hour talk) — the best companion to Chapters 2 and 4
- Yao Fu et al.: How does GPT obtain its ability? — the GPT-3 → ChatGPT family tree of Chapter 4
- Scaling Laws for Neural Language Models (2020) · Chinchilla: Training Compute-Optimal LLMs (2022) · Emergent Abilities of LLMs (2022)
- GPT-3: Language Models are Few-Shot Learners · Sennrich et al.: subword units with BPE
- Self-Instruct · Learning to summarize from human feedback · InstructGPT
- SFT Memorizes, RL Generalizes (2025) · Densing Law of LLMs (2024)