1The lecture map
Four acts, eleven chapters. First why efficiency decides who can build and use LLMs; then two ways to change the model (efficient architectures and compression); then two ways to change how we run it: cheaper training and faster serving. Keep the four costs from slide 2 in mind — FLOPs, bytes moved, bytes stored and communication — because every trick attacks one of them. Click a chapter to watch it.
Chapter by chapter what each chapter covers, and the question it leaves for the next
- IWhyDemand grows faster than hardware
- 1
Why Efficiency? Scaling Laws, Costs and the Memory Gap
Scaling laws reward ever more compute, model size grows far faster than GPU memory, and frontier models are expensive to use. The methods split into changes within the Transformer and changes that work on any trained network — and for LLMs, memory and communication join FLOPs as the costs that matter.
→ Where inside a Transformer is the work we could avoid? - IIChange the modelCheaper attention, sparse FFNs, fewer bits, smaller students
- 2
Redundancy and Efficient Attention: Sparse, GQA and MLA
Trained weights are low-rank and attention heads and FFN neurons are decomposable, so there is redundancy to remove. Sparse patterns (Longformer, Big Bird) let each token attend to fewer tokens; MQA, GQA and MLA store fewer keys and values per token.
→ Attention is cheaper — what about the FFN, which holds most of the parameters? - 3
Mixture of Experts: Routing, Experts and DeepSeekMoE
A router sends each token to only a few expert FFNs: the capacity of a huge model at the compute of a small one. Top-k and hash routing, fine-grained and shared experts, load-balancing losses, DeepSeekMoE and DeepSeek-V2, and upcycling a dense model into an MoE.
→ What if the model is already trained, and simply too big? - 4
Quantization: Fewer Bits per Weight
Snap every weight to a small integer grid with a scale and a zero point. Outliers stretch the grid, so LLM.int8() keeps them in FP16, and GPTQ and AWQ make 4-bit work. INT8 is almost free; INT4 is the sweet spot for local models.
→ We kept every weight, just with fewer bits — do we need every weight at all? - 5
Pruning and Knowledge Distillation
Pruning removes weights, heads or whole layers; only structured pruning makes dense hardware faster, and the model needs to recover afterwards, as in LLM-Shearing. Distillation trains a small student on a teacher's soft labels — and for LLMs, mostly on the teacher's generated data.
→ The model is smaller — but how do we afford to train and adapt it? - IIITrain it cheaperTrain fewer parameters; split the rest across GPUs
- 6
Parameter-Efficient Fine-Tuning: LoRA, Adapters, Modules
Freeze the pre-trained model and learn a thin update: LoRA's B·A trains about 0.4% of a 4096×4096 matrix and merges back with zero latency. Adapters and prefix-tuning insert parameters elsewhere, and small modules might be composed like Lego bricks.
→ What if the model itself does not fit on one GPU? - 7
Distributed Training: Data and Model Parallelism, ZeRO
Data parallelism copies the model and splits the data; model parallelism splits the model. Mixed-precision Adam needs 16 bytes per parameter — 112 GB for a 7B model — and ZeRO shards optimizer states, gradients and weights so memory per GPU falls with the number of GPUs.
→ Training happens once — inference runs billions of times. What makes it slow? - IVServe it fasterFewer bytes per token, more tokens per pass
- 8
Inference Bottlenecks, the KV Cache and PagedAttention
Prefill is compute-bound, decoding is memory-bound. The KV cache grows with every token and, not the weights, limits the batch size; PagedAttention stores it in small blocks like virtual memory, cutting the waste from 60–80% to under 4%.
→ Memory is under control — but each request still makes one big pass per token. - 9
Speculative Decoding: Draft, Verify, Stay Exact
A small draft model proposes several tokens and the large model checks them all in one pass. Accept with probability min(1, q/p), otherwise resample from the normalized max(0, q−p): the output distribution is exactly the big model's, typically 2–3× faster.
→ Fewer big passes — but where does the time go inside one pass? - 10
FlashAttention: Tiling and the Online Softmax
Standard attention writes the N×N score matrix to slow HBM and reads it back. FlashAttention computes attention tile by tile in on-chip SRAM, using a running max and sum to merge softmax blocks exactly: more FLOPs, 9× fewer bytes moved, 5.7× faster.
→ We have all the building blocks — how does a real serving engine combine them? - 11
Serving with vLLM and the Road Ahead
vLLM looks like an operating system: a scheduler with continuous batching, a block manager with page tables, copy-on-write sharing for parallel sampling and beam search, and preemption. The outlook: sparse and hybrid attention, low-bit formats, disaggregated serving and efficient reasoning.
2Try it in Colab
Three short notebooks turn the lecture's arithmetic into experiments. Click Open in Colab, then Runtime → Run all. Each ends with exercises. All three run on the free CPU runtime with small models (SmolLM2-135M, GPT-2); no GPU needed.
The KV cache: a memory calculator and GQA
- Reproduce slide 117: one 4K-token LLaMA-2-70B request needs 10.7 GB of KV cache with MHA, 1.34 GB with GQA
- Read seven open models' configs: GQA saves 4–8×, DeepSeek's MLA about 57×
- Qwen2.5-7B on an 80 GB GPU at 8K context: 137 requests fit instead of 19
- Measure a real cache, time decoding with and without it, and see why sharing K/V heads after training breaks GPT-2
.ipynb · CPUQuantize a small model and watch its perplexity
- Watch one outlier round every other INT4 weight to zero
- SmolLM2-135M on WikiText-2: INT8 costs +0.4% perplexity; INT4 goes from broken (one scale per tensor) to +39% (groups of 64)
- Find activation outliers above 2,000 and keep them in full precision, as LLM.int8() does
.ipynb · CPULoRA parameter counts and training memory
- LoRA rank 8 on a 4096×4096 matrix: 65,536 parameters, 0.39%
- Redo the memory bill for a 7B model: 112 GB for full fine-tuning, ~15 GB with LoRA, ~4 GB with QLoRA (before activations)
- Reproduce ZeRO's 120 / 31.4 / 16.6 / 1.9 GB per GPU
- Write LoRA from scratch, train it on CPU, and merge B·A back into W
.ipynb · CPU3Concept index
Key terms of the lecture. Each link opens the video at the slide where the idea is explained.
4What's new in 2026
The lecture's outlook (slide 165) asks where the next ten-times saving will come from. Its answer: the frontier is co-design — model architecture, numerics, kernels and serving systems built together. Six trends to watch:
Sparse attention in production
DeepSeek-V3.2 adds a lightweight “lightning indexer” that scores past tokens and lets each query attend only to the top-k of them (k = 2,048) — Longformer's idea (Chapter 2), but learned.
Hybrid linear attention
Most layers become linear-attention layers with a fixed-size state; only every fourth layer keeps a growing KV cache. Kimi Linear reports up to 75% less KV cache and up to 6× decoding throughput at 1M tokens.
Low-bit everything
FP8 training is now standard — DeepSeek-V3 was trained largely in FP8 (slide 55). NVFP4 on NVIDIA Blackwell stores 4-bit floats with one scale per block of 16 values: the group-wise quantization of Notebook 2, in hardware.
Disaggregated serving
Compute-bound prefill and memory-bound decode (slide 114) now run on separate GPU pools, with the KV cache moved between them over fast networks. Mooncake serves Kimi this way; llm-d brings it to Kubernetes on top of vLLM.
Efficient reasoning
For reasoning models, thinking tokens dominate the bill. Thinking budgets, adaptive effort, multi-token prediction and speculative decoding (Chapter 9) all aim at cheaper long chains of thought.
5Quotes & reading
"We should forget about small efficiencies, say about 97% of the time: premature optimization is the root of all evil. Yet we should not pass up our opportunities in that critical 3%."
The lecture's method: first ask where the time and memory go, then optimize the critical part.
"The biggest lesson that can be read from 70 years of AI research is that general methods that leverage computation are ultimately the most effective, and by a large margin."
If quality is bought with compute (slide 6), every saving buys a better model.
"We argue that a missing principle is making attention algorithms IO-aware — accounting for reads and writes between levels of GPU memory."
Bytes moved, not FLOPs — the lesson of Chapter 10.
"Many insects have a larval form that is optimized for extracting energy and nutrients from the environment and a completely different adult form that is optimized for the very different requirements of traveling and reproduction."
Train big, deploy small: the idea behind distillation and compression (Chapters 4–5).
Further reading
- Song Han (MIT): 6.5940 TinyML and Efficient Deep Learning Computing — the course this lecture draws on most (slide 168), with slides and videos
- Lilian Weng: Large Transformer Model Inference Optimization (2023) — a single overview of quantization, pruning, distillation and sparsity
- Architectures: GQA (2023) · DeepSeek-V2 and MLA (2024) · Switch Transformers (2021) · DeepSeekMoE (2024)
- Compression: LLM.int8() · GPTQ · AWQ · Sheared LLaMA · Distilling the Knowledge in a Neural Network
- Training: LoRA (2021) · QLoRA (2023) · ZeRO (2019)
- Inference: FlashAttention (2022) · PagedAttention / vLLM (2023) · Speculative decoding · Speculative sampling (2023)
- Anyscale: How continuous batching enables 23x throughput in LLM inference (2023) · code: vLLM, FlashAttention, PEFT, DeepSpeed