1The lecture map
Five acts, nine chapters. First why next-word prediction is so powerful, then the narrow sense: count n-grams and generate with them, fix and measure them with smoothing and perplexity. Then the broad sense: learn representations with neural LMs, word vectors, ELMo and BERT, and finally choose an architecture — encoder, encoder-decoder or decoder — and see why decoders took over. Click a chapter to watch it.
Chapter by chapter what each chapter covers, and the question it leaves for the next
- IWhyWhat an LM is, and why next-word prediction teaches so much
- 1
Language Models in a Nutshell, and Why They Matter
A language model scores how likely a text is, or equivalently predicts the next word; a neural LM generates "我思故我在" one token at a time. Because the text is its own label, the task scales, and filling blanks forces the model to learn arithmetic, sentiment, coreference and world knowledge.
→ What exactly is "the probability of a sentence"? - IICountBuild an LM from probability and counts, then let it write
- 2
Probability Foundations: Distributions and the Chain Rule
A bag of colored shapes, then the Alice passage as a bag of words: sampling, distributions, conditional probability, and the chain rule that turns a sentence probability into a product of next-word predictions.
→ The full history is too long to count — what can we assume instead? - 3
N-gram Language Models: Counting and Estimation
The Markov assumption keeps only the last n−1 words. Unigram and bigram models for Alice, estimated by relative frequency (MLE), become proper language models once we add EOS, UNK and BOS.
→ What kind of text does such a model write? - 4
Generating Text: Spinning the N-gram Wheel
Sampling is spinning a roulette wheel whose slices are the next-word probabilities. The bigram wheel writes "the book her sister was reading"; the trigram wheel has a single slice almost every time and mostly copies the passage.
→ Bigger n is more fluent — what does it cost? - IIIFix & measureUnseen events, smoothing, perplexity and WER
- 5
Sparsity, Smoothing and Evaluation: Perplexity and WER
Large n copies the corpus, and 99.96% of possible Shakespeare bigrams never occur, so MLE gives most sentences probability zero. Smoothing moves mass to unseen events; perplexity (computed in log space) and word error rate tell us which model is better.
→ Counts cannot share statistics between similar words — can a neural network? - IVLearnFrom counts to vectors, from static to contextual
- 6
From N-grams to Neural LMs and Word Vectors
N-grams suffer from sparsity and storage. The fixed-window neural LM (Bengio et al., 2003) embeds the context instead of counting it, and its embeddings lead to word vectors and word2vec; subword models handle words the vocabulary has never seen.
→ One vector per word — but which "bank"? - 7
Contextualized Word Vectors: ELMo and BERT
ELMo reads with two LSTM language models; BERT uses a bidirectional Transformer trained with masked LM and next sentence prediction, then fine-tunes all weights. A three-step walk-through follows one example from data to inference.
→ How else can we pretrain — and which architecture scales? - VScaleThree architectures, and why the decoder won
- 8
Encoders, Encoder-Decoders and Their Trade-offs
Encoder variants (RoBERTa, SpanBERT, ALBERT, ELECTRA) and encoder-decoders (T5's text-to-text span corruption, BART's denoising), then a side-by-side table of the three families — including the KV cache — and DeepSeek-V4.1-Flash's causal encoder-decoder.
→ Back to plain next-word prediction — at scale. - 9
Back to Next-Word Prediction: GPT, LLaMA and Beyond
Pretrained decoders as classifiers and as generators, GPT-2's unicorn story, GPT-3's in-context learning and open LLaMA models — ending with the two questions every student should be able to answer.
2Try it in Colab
Three short notebooks turn the lecture into experiments. Click Open in Colab, then Runtime → Run all. Each ends with exercises. All three run on a free CPU runtime; no GPU is needed.
N-gram LMs: counting, sampling, smoothing, perplexity
- Reproduce the Alice counts and replay the spin-the-wheel runs with the slides' random numbers
- Generate Shakespeare with 1- to 4-grams and measure how much is copied
- Watch MLE give most test lines probability zero
- Compare add-k smoothing and interpolation by perplexity
.ipynb · CPU, pure PythonA fixed-window neural LM
- Build Bengio et al.'s model in 15 lines of PyTorch
- Train it on the same Shakespeare split and compare perplexity with a trigram model
- Sample with different temperatures
- Inspect the word embeddings it learned along the way
.ipynb · CPU (GPU optional)Sampling from GPT-2, filling BERT's blanks
- Score the monkey's string, real and shuffled text with GPT-2
- Run the autoregressive loop by hand, with and without the KV cache
- Sample with temperature, top-k and top-p
- Fill
[MASK]with DistilBERT and compare the vectors of "bank"
.ipynb · CPU or GPU · ~0.8 GB download3Concept index
Key terms of the lecture. Each link opens the video at the slide where the idea is explained.
4What's new in 2026
The lecture ends with the architecture trade-offs (slide 143) and a September 2026 release (slide 144). Six things to watch around this lecture's topic:
Decoder-only is still the default
One simple objective on raw text, one text-in/text-out interface, cheap generation with the KV cache and predictable scaling: the deck's four reasons why almost all large models are decoder-only.
Encoder-decoder is back — causally
DeepSeek-V4.1-Flash splits 40 layers into a 20-layer causal encoder and a 20-layer decoder that reads one shared KV cache, about a quarter of V4-Flash's KV memory. The name is back; the bidirectional T5 recipe is not.
N-grams return, as memory
DeepSeek's Engram adds hashed n-gram embeddings that a Transformer looks up in constant time and gates into its hidden state. V4.1-Flash uses it with n-gram orders 2–4 and 196B Engram parameters next to its 552B backbone.
Not left to right: diffusion LMs
Diffusion language models write a whole block at once and refine it, a relative of BERT's masked-word prediction. DiffusionGemma, fine-tuned from Gemma 4, denoises 256-token blocks and can still decode autoregressively.
Encoders still earn their keep
BERT-style models remain the tool for understanding, retrieval and embeddings. ModernBERT brought the encoder up to date: trained on 2 trillion tokens with a native 8,192-token context.
A language model helped teach this lecture
The spin-the-wheel animation was built with Claude Opus 5.5 from the sampling slide, the Alice passage and a short instruction in Chinese — a model that itself writes by sampling the next token, just with a far better distribution.
5Quotes & reading
"You shall know a word by the company it keeps."
Distributional semantics: the idea behind word2vec and every embedding since.
"The fundamental problem of communication is that of reproducing at one point either exactly or approximately a message selected at another point."
Text as a choice among possible messages — the probabilistic view of language. The same paper already generated English from word n-gram approximations, as in Chapter 5.
"But it must be recognized that the notion of ‘probability of a sentence’ is an entirely useless one, under any known interpretation of this term."
Norvig reads it as: a sentence never seen before must get probability zero. Smoothing (slide 66) and neural LMs are the answer.
Further reading
- Jurafsky & Martin, Speech and Language Processing (3rd ed. draft): Chapter 3, N-gram Language Models — the best companion to Chapters 2–5
- A Neural Probabilistic Language Model (Bengio et al., 2003) · word2vec (2013)
- ELMo (2018) · BERT (2018) · RoBERTa (2019) · ELECTRA (2020)
- T5 (2019) · BART (2019) · YOCO: You Only Cache Once (2024)
- GPT-2 (2019) · GPT-3: Language Models are Few-Shot Learners (2020) · LLaMA (2023)
- Infini-gram: scaling unbounded n-gram LMs to a trillion tokens (2024) · nanoGPT — train a small GPT yourself