1The lecture map
Five acts, ten chapters. The first half climbs the levels of language — characters, words, subwords, sentences, discourse — and asks where that structure went in today's LLMs. The second half turns words into vectors: first by counting contexts, then by predicting them with word2vec. The thread through both halves is context. Click a chapter to watch it.
Chapter by chapter what each chapter covers, and the question it leaves for the next
- IStructureLanguage has structure below the surface
- 1
What Is Linguistics? Structure, Grammar and Englishes
Linguistics is the scientific study of language. Substitution tests on "The cat sat on the mat" reveal constituents, "Colorless green ideas sleep furiously" separates form from meaning, and English varieties remind us that communication is cooperative.
→ Structure exists at many levels — what are the smallest units a computer actually stores? - IIUnitsFrom bytes and characters to words and subword tokens
- 2
From Characters to Words: Scripts, Unicode, Tokens
Chinese characters carry sound and meaning; encodings from GB2312 to Unicode and UTF-8 map them to bytes. Spaces do not define words, normalization is a trade-off, and Zipf's long tail guarantees out-of-vocabulary words.
→ How can a model represent a word it has never seen? - 3
Morphemes, Chinese Words and Byte Pair Encoding
Words are built from morphemes; Chinese words have little inflection and ambiguous boundaries (乒乓球拍卖完了). GPT-style models sidestep both with byte-level BPE, which learns frequent pieces and can always fall back to bytes. Parts of speech give words a grammatical role.
→ Words have categories — how do they combine into sentences? - IIISentences & beyondWho did what to whom — and what LLMs learn of it
- 4
Sentences and Discourse: Parse Trees to Coherence
Phrase-structure trees group words into constituents; dependency trees link each word to its head. Ambiguity ("I shot an elephant in my pajamas") makes parsing hard, and discourse relations connect sentences into a coherent text.
→ Do modern language models still need all this annotation? - 5
Linguistics Meets Deep Learning: The Pendulum
A review with a contrastive "but" once needed a pipeline of separate analysis stages; a language model trained only to predict text handles it end to end, and probes find linguistic structure inside it. The field swings between empiricism and rationalism — may it swing back?
→ If we describe a word with a short list of numbers, what should the numbers tell us? - IVCountTurn where words occur into vectors
- 6
Words for Search: Term Matching, TF-IDF and LSA
Search needs a score, not just a match. The vector space model with TF-IDF weights ranks documents; latent semantic analysis compresses the term-document matrix with SVD so related words meet even without exact matches.
→ Can the same counting represent individual words? - 7
Distributional Semantics: Context Becomes Vectors
Hand-built resources like WordNet are costly; the distributional hypothesis lets text do the work. You can guess "ongchoi" from its contexts, and so can a computer: co-occurrence counts, cosine similarity, PPMI weighting, and SVD for dense vectors.
→ Instead of counting and factorizing, can we learn dense vectors by prediction? - VPredictLearn dense vectors by predicting context
- 8
Word Embeddings and the Word2vec Skip-Gram Model
Embedding spaces capture neighbors, analogies (king − man + woman), cross-lingual structure, semantic change — and social bias. Skip-gram learns them by predicting context words with a softmax over the vocabulary, using two tables of 2|V|d parameters.
→ Every update sums over the whole vocabulary — how can training be cheap? - 9
Negative Sampling, GloVe and Evaluating Embeddings
Negative sampling turns prediction into a few binary decisions, touching K + 2 vectors per update. CBOW flips the direction, GloVe fits global counts, pretrained vectors save training, and intrinsic and extrinsic tests answer different questions.
→ Where exactly does the full-softmax cost come from? - 10
Wrap-Up and Appendix: Deriving Word2vec Gradients
A recap of the thread from structure to representation, then the calculus: the center-word gradient is the expected context vector minus the observed one, and the full-softmax update costs about |V|·d per pair, against K·d with negative sampling.
2Try it in Colab
Three short notebooks turn the lecture into experiments. Click Open in Colab, then Runtime → Run all. Each ends with exercises. All three run on the free CPU runtime; no GPU is needed.
From bytes to tokens
- See 自然语言处理 as code points, UTF-8, UTF-16 and GBK bytes — and as mojibake
- Draw Zipf's law and the long tail of a novel
- Watch a segmenter split 乒乓球拍卖完了 two ways
- Write BPE from scratch, then compare GPT-2 (about 2 tokens per Chinese character) with Qwen2.5 (自然 | 语言 | 处理)
.ipynb · CPUFrom counts to vectors: TF-IDF, LSA and PPMI
- Build a TF-IDF search engine by hand
- Let LSA match titles that share no word with the query
- Count co-occurrences in Wikipedia; compare cosine with the dot product
- PPMI + SVD: coffee finds milk, red finds blue
.ipynb · CPUWord2vec from scratch, and how to evaluate it
- Time full softmax against negative sampling
- Train skip-gram with negative sampling in PyTorch on 1.7 M words
- Neighbors and analogies with your vectors and with pretrained GloVe
- Score both on WordSim-353 and the Google analogies; probe for gender bias
.ipynb · CPU3Concept index
Key terms of the lecture. Each link opens the video at the slide where the idea is explained.
4What's new in 2026
Word2vec and GloVe date from 2013–2014, and BPE entered NLP in 2016. The questions in this lecture — what a token should be, and what a vector should capture — are still moving. Five developments to watch:
Bigger, more multilingual vocabularies
GPT-2's byte-level BPE vocabulary (the idea on slide 61) has 50,257 tokens. Llama 3 moved to 128K tokens and reports up to 15% fewer tokens than Llama 2; Gemma 3 uses 262K, chosen to be more balanced for non-English text — the cost gap that slide 58 shows for Chinese.
Tokens that cross the space bar
Spaces do not define words ("New York"). SuperBPE lets tokens span whitespace: at a fixed 200K vocabulary it encodes text with up to 33% fewer tokens, and an 8B model trained with it gains 4.0 points on average over 30 tasks.
Tokenizer-free models are catching up
Instead of a fixed BPE vocabulary, models read raw bytes and learn their own grouping: BLT forms patches by next-byte entropy, Bolmo converts an existing Olmo 3 model to bytes, and ByteFlow learns segmentation by compression.
Static embeddings are back — for speed
A lookup table plus averaging, like the baseline on slide 176, trained with modern objectives: Sentence Transformers' static models run 100–400× faster on CPU and keep at least 85% of the quality of transformer encoders.
Shared concepts across languages, inside LLMs
Cross-lingual structure (slide 152) shows up inside large models too: interpretability work found language-independent features for concepts such as "small" and "opposite", and the shared fraction grows with model scale.
5Quotes & reading
"You shall know a word by the company it keeps."
The one-line summary of Chapters 7–9.
"Difference of meaning correlates with difference of distribution."
The distributional hypothesis (slide 132) stated as a testable claim.
"Colorless green ideas sleep furiously."
Grammatical form and meaning are different things.
"The meaning of a word is its use in the language."
The philosophical root of learning meaning from usage.
Further reading
- Jurafsky & Martin: Speech and Language Processing (3rd ed. draft) — the chapters on words and tokens and on vector semantics cover Chapters 2–9
- Efficient Estimation of Word Representations in Vector Space (word2vec) (2013) · Distributed Representations of Words and Phrases (negative sampling, 2013) · GloVe (2014)
- Levy, Goldberg & Dagan: Improving Distributional Similarity with Lessons Learned from Word Embeddings (2015) — why PPMI + SVD and word2vec are closer than they look
- Sennrich, Haddow & Birch: Neural Machine Translation of Rare Words with Subword Units (BPE, 2016) · Andrej Karpathy: Let's build the GPT Tokenizer (video, 2024)
- Kenneth Church: A Pendulum Swung Too Far (2011) — the empiricism–rationalism curve on slide 99 · Manning et al.: Emergent Linguistic Structure in Artificial Neural Networks Trained by Self-Supervision (PNAS 2020)
- Bolukbasi et al.: Man is to Computer Programmer as Woman is to Homemaker? (2016) · Hamilton et al.: Diachronic Word Embeddings Reveal Statistical Laws of Semantic Change (2016) · Universal Dependencies