1The lecture map
Five acts, eleven chapters. Every encoder needs two operations: composition (letting tokens exchange information) and abstraction (transforming the features of each token). MLPs, CNNs and RNNs mix the two and pay with fixed windows, local reach or long, fragile paths. The Transformer separates them — attention composes, feed-forward layers abstract — and it keeps improving as it scales. Click a chapter to watch it.
Chapter by chapter what each chapter covers, and the question it leaves for the next
- IThe problemWhat a contextual encoder must do
- 1
NLP Tasks, Pre-training and Why Transformers
From bigram counts to masked language models and pre-train-then-fine-tune. Four task interfaces — one label per sentence, a tag per character, a start–end span, and token-by-token generation — all sit on one contextual encoder.
→ Every task needs context — how should a network combine the words? - 2
Inductive Bias: Semantic Composition and Abstraction
Abstraction transforms features within a token; composition lets tokens interact. Local windows, recurrent states and global attention are different inductive biases, and the decisive word can lie far away.
→ How do concrete architectures implement composition? - IIBefore attentionWindows, filters and recurrent states
- 3
Multilayer Perceptrons and Neural Language Models
Concatenate word embeddings, apply fully-connected layers, learn from the prediction error by backpropagation — one worked gradient step lifts P(Positive) from 0.40 to 0.81. Bengio's neural LM uses the same recipe over a fixed window.
→ A fixed input layout cannot handle variable lengths — can we reuse one filter at every position? - 4
Convolutional Networks and ConvS2S
Convolution multiplies a window by a filter and sums, sliding over 2D images or 1D token sequences. Causal convolutions generate text by predict-append-repeat; ConvS2S trains all positions in parallel with teacher forcing.
→ A window only sees its neighbours — can a state carry the whole past? - 5
Recurrent Networks: RNN, LSTM and GRU
An RNN reuses one update to carry a hidden state through time, so any length works — but steps run one after another. Teacher forcing trains on gold prefixes; LSTM gates keep, write and expose memory, and the GRU resets, then blends.
→ Can recurrence be made efficient again — and why do distant words still fade? - 6
Mamba, RWKV and Gradient Flow Through Time
Gates compared, deeper and bidirectional RNNs, and the modern recurrent revival: Mamba's selective state updates and RWKV's recurrent inference. Backpropagation through time multiplies Jacobians, so gradients vanish or explode; residual connections give them a direct path.
→ What if every token could reach every other token in one step? - IIIThe TransformerAttention composes, feed-forward abstracts
- 7
The Transformer Architecture: Encoder and Decoder
A tour of the original model: position embeddings added to word embeddings, Add & Norm around every sublayer, a per-token feed-forward network, N stacked blocks, masked self-attention in the decoder and cross-attention to the encoder's memory.
→ What exactly does the self-attention box compute? - 8
Self-attention, Multi-head Attention and Positions
Queries, keys and values; scores 112 and 96 become weights 0.88 and 0.12 after scaling and softmax; several heads run in parallel. Order comes from sinusoidal encodings, from rotating queries and keys (RoPE), or from recurrent layers in hybrids such as Kimi Linear.
→ Full attention compares all pairs — what does that cost, and which parts really matter? - IVBeyondWhat changes, what matters, what scale buys
- 9
Transformer Variants, Applications and Ablations
Attention is quadratic in length; Linformer compresses, Longformer and Big Bird sparsify. The same block runs on image patches, diffusion latents, robot actions and music. Ablations test what each part does: pure attention loses rank, MLP-Mixer and FNet mix tokens without attention.
→ If the architecture scales, how predictable is the gain from scale? - 10
Scaling Laws and Emergent Abilities
Loss falls smoothly with model size, data and compute, well enough to predict a large run from small ones. The budget decides the best model size; test-time compute and agents are new axes. Emergent abilities may partly reflect the choice of metric.
→ Appendix: how are all these gradients actually computed? - VAppendixThe chain rule, worked by hand
- 11
Appendix: Backpropagation Worked Examples
Upstream gradient times local derivative, node by node: a small graph, a sigmoid built from elementary operations, the module interface in PyTorch, and three ways gradients fail — saturation, dying ReLUs and explosion.
2Try it in Colab
Three short notebooks turn Chapters 7–8 into experiments. Click Open in Colab, then Runtime → Run all. Each ends with exercises. All three run on the free CPU runtime; no GPU needed.
Self-attention from scratch
- Reproduce slide 93: scores 112 and 96 become 0.88 and 0.12
- See why scores are divided by √dk
- Write single- and multi-head attention and match PyTorch exactly
- Shuffle the input, add a causal mask
- Train one attention layer on a long-range task: 100% accuracy with positions, about 19% without
.ipynb · CPU, no downloadsPositional encodings: sinusoids and rotations
- From a binary counter to the sinusoidal encoding
- Redraw slide 108 and print the table of slide 109
- Check that a shift is a rotation and similarity depends on distance
- Rebuild the RoPE example of slide 110 and verify it
.ipynb · CPU, no downloadsInside a pretrained Transformer
- Compare the attention masks of DistilBERT (encoder) and GPT-2 (decoder)
- Measure where who attends in the slides' sentence — and test it with a control
- Find previous-token heads in GPT-2
- Look at GPT-2's learned position embeddings
- Time the quadratic cost of attention
.ipynb · CPU, downloads ~0.8 GB3Concept index
Key terms of the lecture. Each link opens the video at the slide where the idea is explained.
4What's new in 2026
The lecture already points at the newest designs (Kimi Linear, slide 111; new scaling directions, slide 144). Six trends to watch:
Hybrid attention goes mainstream
Instead of full attention everywhere, several open models use cheap linear-attention layers (Gated DeltaNet, Kimi Delta Attention) for most of the stack and keep about one full-attention layer in four.
Recurrence, reinvented
Mamba and RWKV keep a fixed-size state instead of a growing cache of keys and values, so inference work grows linearly with length — the recurrent idea of Chapter 5, revisited with learned, input-dependent updates.
Position moves into the layers
RoPE rotates queries and keys rather than adding a vector; Kimi Linear goes further and uses no explicit position encoding in its full-attention layers, letting the recurrent layers carry order.
Sparse attention in production
DeepSeek Sparse Attention adds a small "lightning indexer" that picks which earlier tokens each query attends to — a learned version of the sparse patterns on slide 116, aimed at long contexts.
Attention across depth
Attention Residuals replace the fixed sum of a residual stream with softmax attention over earlier layers, so each layer chooses which earlier layers to read — attention over depth instead of over time.
Scaling beyond parameters
Compute spent at test time (sampling, longer reasoning, search), during RL training, on tools, memory and agents is now measured as its own axis — with gains that depend on how good the selection signal is.
5Quotes & reading
"We propose a new simple network architecture, the Transformer, based solely on attention mechanisms, dispensing with recurrence and convolutions entirely."
The sentence that ended Act II — and the diagram of Chapter 7.
"The biggest lesson that can be read from 70 years of AI research is that general methods that leverage computation are ultimately the most effective, and by a large margin."
Why a weaker inductive bias can win once data and compute grow.
"The loss scales as a power-law with model size, dataset size, and the amount of compute used for training, with some trends spanning more than seven orders of magnitude."
The empirical basis of Chapter 10.
"…emergent abilities appear due to the researcher's choice of metric rather than due to fundamental changes in model behavior with scale."
The counterpoint on slide 147: read capability curves as carefully as loss curves.
Further reading
- Attention Is All You Need (2017) · Jay Alammar: The Illustrated Transformer (the figures of Chapter 8) · Harvard NLP: The Annotated Transformer
- Christopher Olah: Understanding LSTM Networks · Mamba · RWKV — Chapters 5–6
- Amirhossein Kazemnejad: Transformer Architecture: The Positional Encoding · RoFormer (RoPE) · On Position Embeddings in BERT
- Lilian Weng: The Transformer Family Version 2.0 — a survey of the variants of Chapter 9 · Kimi Linear (2025)
- Scaling Laws for Neural Language Models · Training Compute-Optimal LLMs (Chinchilla) · Emergent Abilities of LLMs · … a Mirage?