Tutorial 2: Training Your Own LLMs

1Overview

Lecture 5 explained how LLMs are pre-trained; Lecture 6 explained why post-training uses reinforcement learning. This tutorial runs the whole post-training toolbox at small scale. The slides walk through SFT, parameter-efficient fine-tuning (LoRA, QLoRA) and the three feedback-based methods. The notebooks then train real models: each one isolates one way of turning a training signal into better behaviour.

Learning objectives

• Explain the SFT loss and why only response tokens are graded; estimate GPU memory for full fine-tuning, LoRA and QLoRA.
• Explain why LoRA's update \(\Delta W = BA\) starts at zero, and how QLoRA stores the frozen base in 4-bit NF4.
• Train a reward model from human labels and use it, together with a KL penalty, as the reward for PPO.
• Follow one PPO iteration: rollouts, rewards, GAE advantages from a value head, clipped policy updates.
• Compute GRPO's group-relative advantage by hand and recognise when a group gives no learning signal.
• Write the DPO loss, interpret its training metrics, and say when DPO is preferable to online RL.
• Read training diagnostics critically: tell a real improvement from noise on a small evaluation set.

Time plan

2Slides: Training Your Own LLMs

52 slides in four parts — the big picture, SFT, efficient fine-tuning (LoRA & QLoRA), and learning from feedback (PPO, GRPO, DPO) — with a running example: turning a 7B base model into a helpful, safe medical assistant.

Slide 2 · Today's tutorial
Slide 10 · The SFT loss
Slide 14 · GPU memory for a 7B model
Slide 20 · LoRA: low-rank adaptation
Slide 33 · QLoRA
Slide 40 · One PPO iteration
Slide 45 · Group-relative advantage
Slide 47 · The DPO loss
Slide 51 · Summary: choosing a method

Key ideas from the slides

3Before you start

pip install "transformers==4.51.3" \
    "trl==0.18.2" "peft==0.15.2" \
    "datasets==3.5.0" "accelerate==1.6.0" \
    bitsandbytes math-verify

4Notebook A · SFT, reward model and PPO (RLHF)

4.1 One frozen 4-bit base, three LoRA adapters

Instead of loading three copies of a 4B model, the notebook loads one base in 4-bit NF4 and attaches three independent LoRA adapters (rank 8 on q,k,v,o projections). Switching the active adapter switches the “model”:

qconfig = BitsAndBytesConfig(load_in_4bit=True, bnb_4bit_quant_type="nf4",
                             bnb_4bit_use_double_quant=True, bnb_4bit_compute_dtype=DTYPE)
base = AutoModelForCausalLM.from_pretrained(MODEL_ID, quantization_config=qconfig, device_map={"": 0})
base = prepare_model_for_kbit_training(base, use_gradient_checkpointing=False)

lora = LoraConfig(r=8, lora_alpha=16, lora_dropout=0.0,
                  target_modules=["q_proj", "k_proj", "v_proj", "o_proj"], task_type="CAUSAL_LM")
model = get_peft_model(base, lora, adapter_name="sft")    # adapter 1
model.add_adapter("reward", copy.deepcopy(lora))           # adapter 2
model.add_adapter("ppo", copy.deepcopy(lora))              # adapter 3
reward_head = nn.Linear(H, 1)   # score from the last token's hidden state
value_head  = nn.Linear(H, 1)   # V(s_t) for every generated token

The SST-2 data are split once and frozen, so every stage sees different reviews:

4.2 Component 1 — SFT: learn the domain

SFT continues next-token training on real movie reviews (128 optimizer steps × 8 accumulated reviews, lr 1e-4). It teaches the style of reviews, not positivity: half of the reviews are negative.

The base model drifts into web-text habits (a capitalization quiz); after SFT the continuations look like SST-2 reviews.

4.3 Component 2 — Reward model: learn what “good” means

The reward model is the same backbone with its own adapter and a linear reward_head on the last token. It is trained with binary cross-entropy on human sentiment labels:

4.4 Component 3 — PPO: optimise the reward, stay close to SFT

The reward model's score is mapped to \([-1, 1]\) and added at the last token; every token also pays a KL penalty for drifting from the SFT policy:

A value head estimates \(V(s_t)\); advantages come from GAE (\(\gamma=1,\ \lambda=0.95\)), and the policy is updated with the clipped objective (\(\epsilon=0.2\)):

# --- rollout: per-token KL penalty, reward-model score added at the last token
score   = 2 * reward_logits(sequence).sigmoid() - 1          # in [-1, 1]
rewards = -kl_beta * (old_logprobs - ref_logprobs) * mask
rewards[-1] += score

# --- GAE, computed backwards over the generated tokens
for t in reversed(range(T)):
    delta = rewards[:, t] + gamma * values[:, t + 1] - values[:, t]
    carry = delta + gamma * lam * carry
    advantage[:, t] = carry
returns = advantage + values
advantage = (advantage - advantage.mean()) / advantage.std()  # normalised

# --- clipped policy loss + clipped value loss
ratio = torch.exp(logprobs - old_logprobs)
pg = torch.maximum(-ratio * adv, -ratio.clamp(1 - clip, 1 + clip) * adv).mean()
vclip = old_values + (vpreds - old_values).clamp(-value_clip, value_clip)
vf = 0.5 * torch.maximum((vpreds - returns) ** 2, (vclip - returns) ** 2).mean()
loss = pg + value_coef * vf

4.5 Results and how to read them

Metric (40 test prefixes)	SFT	SFT + PPO
Positive full reviews (external DistilBERT judge)	57.5%	67.5%
Positive continuations only	50.0%	52.5%
Held-out review NLL	3.52	≈ 3.52

5Notebook B · GRPO with verifiable rewards

5.1 Task and verifiable rewards

The policy is Qwen2-0.5B-Instruct with a rank-16 LoRA (2.16M trainable parameters, 0.44% of the model). Training questions are 128 generated arithmetic problems (addition, subtraction with negative results, multiplication, two-step expressions, word problems); 12 fixed questions are held out. The system prompt asks for a short calculation and a final answer in tags:

5.2 Group-relative advantage

For each question GRPO samples a group of \(G=4\) answers and normalises their rewards within the group — the group mean replaces PPO's learned value model:

If all four answers get the same reward, every advantage is zero and the question teaches nothing. The notebook therefore screens training questions first and keeps 8 of the first 12 probes whose answers were mixed (some right, some wrong). Even so, groups often become uniform during training:

training_args = GRPOConfig(
    num_generations=4,            # group size G
    num_iterations=2,             # reuse each sampled group twice (needs the clip)
    max_steps=64, learning_rate=2e-5,
    temperature=0.9, top_p=0.95,  # sampling must be diverse to get mixed groups
    beta=0.02, epsilon=0.2,       # KL coefficient and clip range
    loss_type="grpo", scale_rewards=True,
    reward_weights=[0.1, 1.0],    # format, correctness
)
trainer = GRPOTrainer(model=model, args=training_args,
                      reward_funcs=[format_reward, accuracy_reward],
                      train_dataset=train_dataset, peft_config=lora_config)
trainer.train()

5.3 Results on the 12 held-out questions

6Notebook C · DPO on preference pairs

The data are 128 short pairs from trl-lib/ultrafeedback_binarized (16 more for evaluation). One example:

6.1 The DPO loss

DPO measures how much more the policy prefers the chosen answer than the frozen reference does, and pushes that margin up:

training_args = DPOConfig(
    beta=0.1, loss_type="sigmoid",          # L = -log sigmoid(beta * margin)
    learning_rate=5e-6, max_steps=32,
    per_device_train_batch_size=1, gradient_accumulation_steps=4,
)
trainer = DPOTrainer(model=model, ref_model=None,   # None: TRL makes a frozen copy as reference
                     args=training_args, train_dataset=train_dataset,
                     eval_dataset=eval_dataset, processing_class=tokenizer)
trainer.train()
metrics = trainer.evaluate()   # rewards/accuracies, rewards/margins, loss

6.2 Results: a lesson in reading metrics

With 32 steps at learning rate 5e-6, the loss never leaves \(\ln 2\): the model has barely moved from the reference, and the evaluation margin is indistinguishable from zero. The answers do change (greedy decoding amplifies tiny probability changes) — for example the photosynthesis explanation became a numbered list — but these changes are not evidence of learned preferences.

7Choosing a method

In Notebook A the four PPO “models” share one 4-bit base: they are adapters and small heads, which is why a 4B model fits on a 16 GB T4.

8Practical AI tools for research

A short companion deck by Siqi Rao covers three tools you will need for assignments and the final project.

import os
from openai import OpenAI   # many providers speak the same protocol

client = OpenAI(api_key=os.environ["LLM_API_KEY"],          # never hard-code the key
                base_url="https://api.deepseek.com")         # change for another provider
reply = client.chat.completions.create(
    model="deepseek-chat",
    messages=[{"role": "user", "content": "Explain DPO in two sentences."}])
print(reply.choices[0].message.content)

9Pitfalls & troubleshooting

10Exercises

Not graded — but they map directly onto Assignment 2 and typical final-project experiments.

11Self-check quiz

Click a question to reveal the answer.

12Further reading & credits

Next steps: Lecture 6 slides (RL for LLM post-training) · Tutorial 1 · Course schedule