1The lecture map
Five acts, nine chapters. The model first learns to see, then to hear, speak and create, then to act in a 3D world — and through all of it, language stays the hub that connects the modalities. Click a chapter to watch it.
Chapter by chapter what each chapter covers, and the question it leaves for the next
- IWhyWhy text alone is not enough
- 1
Why Multimodality?
Humans perceive through many senses at once, the web is multimodal, and high-quality text is running out.
→ Can a model perceive the world the way we do? - IISeeMatch images to text, then read them
- 2
Connecting Vision and Language: CLIP and Flamingo
CLIP puts images and text in one space by contrastive learning, which gives zero-shot recognition and retrieval. Flamingo lets a frozen LLM read images through a Perceiver Resampler and gated cross-attention.
→ Matching and few-shot reading work — but how should a general model be built? - 3
Multimodal LLMs: Architecture
Language is the pivot. The LLM either controls expert tools, or vision becomes part of the model (encoder + connector + LLM, as in LLaVA and Qwen-VL), up to native models such as GPT-4o and Gemini.
→ The architecture is set — what data teaches it? - 4
Training Multimodal LLMs
Stage 1 aligns image features with the LLM on captions; stage 2 tunes it on visual instructions, often written with GPT's help.
→ Images work — what about sound? - IIIHear, speak & createTalk in real time; create images and video
- 5
Speech Language Models
Speech is long, continuous and rich in tone. Turn it into tokens, interleave them with text, listen while speaking, and split the work between a thinker and a talker.
→ The model can understand and talk — can it also create? - 6
Generation and World Simulators
Unify understanding and generation, predict the next frame, and use video models as simulators of the world.
→ Is a generated world physically right — and can a model act in it? - IVAct in 3DTurn perception into action in space
- 7
Embodied Intelligence: VLA and World Models
Vision-language-action models map pixels and instructions to motor commands: System 2 plans while System 1 acts, and world-action models predict both the future and the action.
→ Acting needs space — what do pictures miss? - 8
3D Understanding and Generation
Images are only projections. Feed-forward 3D reconstruction, 3D generation, and an LLM that writes scene graphs.
→ What is still open? - VNextWhat is still open
- 9
Research Directions and Outlook
Efficiency, evaluation, more modalities, edge devices — and what changed in 2026.
2Try it in Colab
Three short notebooks turn the lecture into experiments. Click Open in Colab, then Runtime → Run all. Each ends with exercises. Notebook 2 needs a free T4 GPU (Runtime → Change runtime type); the others also run on CPU.
CLIP: zero-shot classification and retrieval
- Classify images into labels CLIP never trained on
- See a prompt raise P(cat) from 0.24 to 0.69
- Search images with text, and find where it fails
- Compute the contrastive loss by hand
.ipynb · CPU or GPUChat with an open multimodal LLM
- Ask Qwen2.5-VL-3B about images, alone or several at once
- Count how many tokens an image becomes
- Find the vision encoder, connector and LLM inside the model
.ipynb · T4 GPUSpeech as tokens
- Compress speech with a neural codec and listen at 1.5–12 kbps
- Check with Whisper whether the words survive
- See that speech needs ~160× more tokens than its transcript
.ipynb · CPU or GPU3Concept index
Key terms of the lecture. Each link opens the video at the slide where the idea is explained.
4What's new in 2026
The lecture's last part (slide 127) lists what changed this year. Six trends to watch:
Omni is the default
Real-time speech-and-vision models are the new baseline, and assistants now listen while they speak and keep working in the background.
World models became products
Interactive world models moved from research demos to products for driving simulation and developers.
From VLA to world-action models
Robot models now predict the future as well as the action, and can be steered by prompts instead of retraining.
General models go physical
Frontier general-purpose models are being evaluated as robot planners and released in robotics editions.
3D in one forward pass
Feed-forward reconstruction and native 3D generation are now standard tools rather than hours of per-scene optimization.
One embedding space for everything
Embedding models now map text, images, video, audio and PDFs into a single space — CLIP's idea, extended to every modality.
5Quotes & reading
"The limits of my language mean the limits of my world."
The case for language as the hub — and for giving it more of the world to describe.
"Language is an imperfect, incomplete, and low-bandwidth serialization protocol for the internal data structures we call thoughts."
Why text alone is not enough.
"If we want machines to think, we need to teach them to see."
The motivation behind ImageNet, and behind Chapters 2–4.
"What I cannot create, I do not understand."
Why generation and world simulators (Chapter 6) test understanding.
Further reading
- Lilian Weng: Generalized Visual Language Models — the best single overview of Chapters 2–3
- CLIP (2021) · Flamingo (2022) · LLaVA: Visual Instruction Tuning (2023)
- Moshi: a speech-text foundation model for real-time dialogue · Qwen2.5-Omni
- OpenAI: Video generation models as world simulators (2024)
- OpenVLA · π0: a vision-language-action flow model · VGGT