CSC5051/CSC6052/MDS5110 Natural Language Processing › Fall 2026 › Lectures

Lecture 6: RL for LLM Post-Training — From Alignment to Reasoning

How did a text predictor become a helpful assistant, then a model that checks its own proofs, and now an agent? One tool did most of the work: reinforcement learning, fed by rewards from people, then from verifiers, then from the world. Watch the lecture in ten chapters with English and Chinese subtitles, follow the lecture map below, then try the ideas in Colab.

📅 Oct. 30th, 2026 🎥 10 chapters · 156 min 🖼 181 slides 💬 EN + 中文 subtitles 🧭 Lecture map 💻 Colab notebooks ⬇ Slides (PDF)
Transcript follows the video · click a sentence to jump · Shift + ←/→ jumps between slides

1The lecture map

Why does reinforcement learning, not more supervised fine-tuning, align LLMs and teach them to reason?

Four acts, ten chapters. Imitation gives a model the format of an assistant; RL on human preferences aligns it. Opening the RL toolbox shows that PPO, ReMax and GRPO differ mainly in their baseline. With rewards that are checked instead of learned, the same tool produces reasoning models, and next it has to teach models to act. The algorithm barely changes; the source of the reward does. Click a chapter to watch it.

IAlign
From imitation to human preferences
1How LLMs are trained2RLHF3DPO & reward hacking
IIThe RL toolbox
Cheaper baselines, checkable rewards
4PPO, REINFORCE, ReMax5GRPO, RLVR & frontier
IIIReason
Long chains of thought, learned by RL
6o1 and o37Measuring & verifying8DeepSeek-R1 & SimpleRL9Self-verification
IVAct
Rewards from tools and environments
10Beyond math & agentic RL
Chapter by chapter what each chapter covers, and the question it leaves for the next
  1. IAlignFrom imitation to human preferences
  2. 1

    How LLMs Are Trained: Pre-Training, Mid-Training and SFT

    Pre-training gives knowledge, mid-training sharpens what post-training will unlock, and SFT teaches the format of an assistant. But SFT sees only positive examples, copies its demonstrator, and optimizes likelihood rather than what people want.

    → Humans can compare answers more easily than write them — how do we train on comparisons?
    Three training stagesMid-trainingPre-training vs. SFTBack-translationWhat SFT cannot do
  3. 2

    Alignment via RLHF: From Summaries to ChatGPT

    Helpful, honest and harmless cannot be written as a loss. RLHF learns a reward model from pairwise comparisons with the Bradley–Terry model, then optimizes the policy with PPO and a KL leash. InstructGPT and ChatGPT scaled this recipe up.

    → Can we learn from the same preference pairs without a reward model and PPO?
    The three H'sOver-refusalPolicy gradientBradley–TerryRLHF in three stepsRLHF beats SFTInstructGPT
  4. 3

    DPO, Biased Judges and Reward Hacking

    DPO turns the RLHF objective into a single classification-style loss on preference pairs. But preference labels are biased (fake references fool every judge), and RL exploits a learned reward: Goodhart's law.

    → What if the reward is checked instead of learned — and which RL algorithm should optimize it?
    DPOOn- vs. off-policyMeasuring judge biasWho is biased?Reward hacking
  5. IIThe RL toolboxCheaper baselines, checkable rewards
  6. 4

    RL for LLMs: PPO, REINFORCE and ReMax

    Text generation is an RL problem with heterogeneous rewards and enormous compute. PPO's critic doubles memory; since LLM feedback is fast and deterministic, REINFORCE with a greedy-answer baseline (ReMax) suffices.

    → If sampling gets cheap, can a whole group of samples give a better baseline?
    Text as an RL problemHeterogeneous rewardsPPOThe price of PPOGeneral RL vs. RL for LLMsReMaxAlgorithms drive infrastructure
  7. 5

    GRPO, RLVR and the Frontier of RL for LLMs

    GRPO uses the group mean as baseline, and its variants (Dr. GRPO, DAPO, GSPO, …) fix stability at scale. Verifiable rewards replace learned ones (RLVR); the frontier moves to asynchronous systems and to exploring hard prompts.

    → What happens when this tool, fed by verifiers, is applied at scale?
    ReMax → GRPOGRPOThe GRPO familyMap of algorithmsSFT memorizes, RL generalizesRLHF vs. RLVRNo gradient on hard promptsKnapsack RLIncentivize, don't teach
  8. IIIReasonLong chains of thought, learned by RL
  9. 6

    Large Reasoning Models: OpenAI o1 and o3

    Long chains of thought outline, plan, backtrack and self-evaluate. o1 improves with both train-time RL and test-time thinking; reasoning tokens make thinking cost real money; o3 thinks with images.

    → How do we measure reasoning fairly?
    Benchmarks shoot upLong CoTo1's thinking habitsTrain- and test-time computeReasoning tokensHacking reasoning tokensThinking with images
  10. 7

    Measuring Reasoning and Inference-Time Verification

    Math, code, GPQA and Humanity's Last Exam are checkable and unsaturated. One path to reasoning: sample many solutions and verify them, guided step by step by a value model (OVM) or a process reward model.

    → Learned verifiers can be hacked — is there a simpler path?
    Why math, code, GPQAGPQAHumanity's Last ExamRepeated samplingOVM decodingProcess vs. outcomeTheorem proving
  11. 8

    Long-CoT Distillation, DeepSeek-R1 and SimpleRL

    Distilling long chains cannot beat the teacher. DeepSeek-R1-Zero showed that pure RL with rule-based rewards makes long reasoning and reflection emerge; SimpleRL reproduced the core result on a 7B model with 8K examples.

    → A right final answer is not a right proof — how do we reward rigorous reasoning?
    Synthesizing long CoTLength emerges in RLThe aha momentR1 pipelineSimpleRL recipeRL vs. SFT on MATHTraining phasesHypotheses
  12. 9

    Self-Verification, Olympiad Gold and ARC-AGI

    DeepSeekMath-V2 trains a proof verifier, checked by a meta-verifier, and rewards the generator with it, reaching open IMO gold. ARC-AGI asks a different question: can a model learn a new skill, or a new game, from a few tries?

    → What about domains with no answer key, and models that must act?
    Why self-verificationGenerator, verifier, meta-verifierIMO, CMO, Putnam20× smaller goldARC-AGIARC-AGI-3
  13. IVActRewards from tools and environments
  14. 10

    RL beyond Math and Toward Agentic RL

    Make medicine and video look like math with verifiable problems and contrastive rewards (HuatuoGPT-o1, Video-R1), or adapt to a domain with RL and a retiring teacher (OnePO). Agentic RL is hard mostly for systems reasons; the reward now comes from the world.

    HuatuoGPT-o1Video-R1 (T-GRPO)HuatuoGPT-3Agentic thinkingWhy agentic RL is hardThe second halfWhy work on RLTakeaways

2Try it in Colab

Three short notebooks turn the lecture into experiments. Click Open in Colab, then Runtime → Run all. Each ends with exercises. Notebooks 1 and 2 run on a CPU; notebook 3 is best on a free T4 GPU (Runtime → Change runtime type) and falls back to a smaller setting on a CPU.

Notebook 1 · Chapters 2–3 · ~5 min

Reward models and DPO from scratch

  • See annotators disagree on scores but agree on comparisons; fit a Bradley–Terry reward model
  • Watch best-of-N hack a length-biased reward model: the proxy rises, true quality peaks and falls
  • Check that DPO lands exactly on the RLHF optimum πref·er/β
  • Write the DPO loss for GPT-2 and train it on eight preference pairs
Open in Colab .ipynb · CPU
Notebook 2 · Chapters 4–5 · ~3 min

GRPO, baselines and pass@k on a verifiable task

  • Train a tiny LM on a sloppy teacher's times tables, then let RL with a verifier beat the teacher
  • Compute GRPO advantages and see why all-wrong groups give no gradient
  • Measure how much ReMax's and GRPO's baselines cut gradient noise
  • Compare pass@k before and after RL
Open in Colab .ipynb · CPU, no downloads
Notebook 3 · Chapters 6–7 · ~10 min

Test-time compute: sample, vote, verify, and “Wait”

  • Solve GSM8K problems with Qwen2.5-0.5B-Instruct and a rule-based answer checker
  • Plot pass@k (a perfect verifier) against majority vote as more tokens are spent
  • Try s1's budget forcing: append “Wait” and see whether answers get fixed
Open in Colab .ipynb · T4 GPU (CPU: smaller run)

3Concept index

Key terms of the lecture. Each link opens the video at the slide where the idea is explained.

Training pipeline
Pre-, mid- and post-trainings4Mid-trainings5 SFT (loss on the answer only)s6Instruction back-translations7 Limits of imitations9
RLHF
Helpful, honest, harmlesss11Over-refusals14 Policy gradients19Reward models21 Bradley–Terry models22KL penalty to the references24 InstructGPTs26ChatGPT (iterated RLHF)s29
Preferences and their limits
DPOs30On-policy vs. off-policys31 Attack success rate (judge bias)s35Authority and beauty biass36 Reward hacking, Goodhart's laws37
RL algorithms
State, action, policy for texts41Baseline and advantages41 Heterogeneous rewardss42PPO clippings46 Critic (value model)s46REINFORCEs50 ReMax (greedy baseline)s52verl / HybridFlows55
GRPO family
Group baselines57GRPOs58 DAPO (clip-higher, dynamic sampling)s64Dr. GRPO, GSPO, CISPOs64 Map of RL algorithmss65
RLVR and the frontier
SFT memorizes, RL generalizess67RL with verifiable rewardss70 Asynchronous RLs74Zero gradient on hard promptss75 Knapsack RLs77Tree-structured rollouts (TreePO)s78 Incentivize, don't teachs80
Reasoning models
Chain-of-thoughts83Long CoT and self-reflections85 Backtrackings92Test-time compute scalings96 Reasoning tokenss97Budget forcing (“Wait”)s98 Thinking with imagess106
Evaluation and verification
GPQAs109Humanity's Last Exams111 Repeated samplings122Best-of-N with a verifiers123 OVM (value-guided decoding)s125Process vs. outcome supervisions127 OmegaPRMs128Lean theorem provings130
R1 and after
Stream of Search, journey learnings132Aha moments137 R1-Zero vs. R1s138Cold start, rejection samplings139 Rule-based rewardss143Length dynamics in RLs148 Verifier and meta-verifiers160ARC-AGIs164
Beyond math and agents
HuatuoGPT-o1s169T-GRPO (Video-R1)s171 OnePO (retiring teacher)s173Agentic thinkings175 Why agentic RL is hards177The second half of AIs178

5Quotes & reading

"When a measure becomes a target, it ceases to be a good measure."

Goodhart's law, in Marilyn Strathern's wording (1997), quoted on slide 37
Why a learned reward model gets hacked once RL optimizes against it.

"Our large-scale reinforcement learning algorithm teaches the model how to think productively using its chain of thought in a highly data-efficient training process."

The o1 recipe in one sentence: RL, chain of thought, data efficiency.

"Rather than explicitly teaching the model on how to solve a problem, we simply provide it with the right incentives, and it autonomously develops advanced problem-solving strategies."

DeepSeek-AI, DeepSeek-R1 (2025), on the “aha moment”
“Incentivize, don't teach” (slide 80), in the authors' own words.

"The biggest lesson that can be read from 70 years of AI research is that general methods that leverage computation are ultimately the most effective, and by a large margin."

Why RLVR, whose signal scales with compute rather than labelling, took over.

Further reading

6About this video

This is an AI-narrated version of the lecture. The slides are the ones used in class. The spoken script was written from the slides with AI assistance and is read in an AI clone of Prof. Benyou Wang's voice (MiniMax speech synthesis); the picture in the corner is a still photo. The red dot points at what is being discussed. When the narration and the slides disagree, trust the slides and the in-class lecture — and please tell us, so we can fix it.
CSC5051/CSC6052/MDS5110 Natural Language Processing · CUHK-Shenzhen · Fall 2026 · Back to the course page