We will preference-tune SmolLM2-135M-Instruct, a small model that has already undergone instruction fine-tuning. The dataset contains two responses to the same prompt: a preferred response (chosen) and a less preferred response (rejected).
DPO trains directly on these fixed offline pairs. It does not train a separate reward model, learn a value head, or generate online rollouts during optimization. The reference policy is a frozen copy of the starting model.
Select GPU T4 x2 (or another CUDA GPU), enable Internet, and start a fresh Kaggle session. Run this cell before importing the training libraries. Only the first GPU is used.
import os
import sys
import json
import subprocess
import importlib
from pathlib import Path
from importlib.metadata import version, PackageNotFoundError
os.environ["CUDA_VISIBLE_DEVICES"] = "0"
os.environ["TOKENIZERS_PARALLELISM"] = "false"
os.environ["HF_HUB_DISABLE_XET"] = "1"
# These are the DPO training dependencies only. Keep Kaggle's CUDA PyTorch.
PINS = {
"transformers": "4.51.3", "trl": "0.18.2", "datasets": "3.5.0",
"accelerate": "1.6.0", "peft": "0.15.2",
"huggingface-hub": "0.36.2", "tokenizers": "0.21.4",
}
ENV_DIR = Path("/kaggle/working/.dpo_env")
SETUP_LOG = Path("/kaggle/working/dpo_setup.log")
fingerprint = {"revision": "private-env-v3-no-ensurepip", "python": sys.version.split()[0],
"executable": sys.executable, "pins": PINS}
marker = ENV_DIR / "dpo_setup_complete.json"
ready = marker.exists() and json.loads(marker.read_text()) == fingerprint
# Imported modules cannot safely be switched to a different version in place.
guard_names = ("transformers", "trl", "datasets", "accelerate", "peft",
"huggingface_hub", "tokenizers", "fsspec", "dill", "multiprocess")
loaded = [name for name in guard_names if name in sys.modules]
if loaded and not globals().get("DPO_PRIVATE_ENV_ACTIVE", False):
raise RuntimeError("Start a fresh Kaggle session and run this cell first. Already loaded: " + str(loaded))
def setup_command(command):
with SETUP_LOG.open("a") as log:
result = subprocess.run(command, stdout=log, stderr=subprocess.STDOUT, text=True)
if result.returncode:
print(SETUP_LOG.read_text()[-16000:])
raise RuntimeError(f"DPO setup failed (exit {result.returncode}). See {SETUP_LOG}.")
if not ready:
SETUP_LOG.parent.mkdir(parents=True, exist_ok=True)
SETUP_LOG.write_text("DPO private environment setup\n")
print("Preparing the private DPO environment; this may take a few minutes.", flush=True)
# Kaggle may not support ensurepip. The host pip can manage a pip-free venv.
if tuple(int(part) for part in version("pip").split(".")[:2]) < (22, 3):
raise RuntimeError("This setup needs host pip >= 22.3 for the --python option.")
setup_command([sys.executable, "-m", "venv", "--without-pip",
"--system-site-packages", str(ENV_DIR)])
env_python = str(ENV_DIR / "bin/python")
# Ignore unrelated shared-app conflict summaries only. Pip failures and the
# trainer's own dependency/import checks remain visible and fail loudly.
setup_command([sys.executable, "-m", "pip", "--python", env_python,
"install", "--disable-pip-version-check",
"--no-warn-conflicts", *[f"{k}=={v}" for k, v in PINS.items()]])
else:
env_python = str(ENV_DIR / "bin/python")
env_site = subprocess.check_output(
[env_python, "-c", "import sysconfig; print(sysconfig.get_paths()['purelib'])"], text=True
).strip()
if env_site in sys.path:
sys.path.remove(env_site)
sys.path.insert(0, env_site)
importlib.invalidate_caches()
def installed_version(name):
try:
return version(name)
except PackageNotFoundError:
return None
actual = {k: installed_version(k) for k in PINS}
assert actual == PINS, f"Unexpected DPO package versions: {actual}"
# This text-only notebook does not need optional vision/audio integrations.
# Blocking these optional imports locally avoids binary mismatches without
# removing torchvision, torchaudio, or librosa from the shared environment.
for optional_name in ("torchvision", "torchaudio", "librosa"):
if optional_name not in sys.modules:
sys.modules[optional_name] = None
from trl import DPOConfig, DPOTrainer
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch
import transformers
import datasets
import accelerate
import peft
DPO_PRIVATE_ENV_ACTIVE = True
marker.write_text(json.dumps(fingerprint, indent=2))
print(actual)
print("PyTorch:", torch.__version__)
print("Transformers loaded from:", transformers.__file__)
print("DPO setup checks passed. Continue to the next cell.")
print("Installation log:", SETUP_LOG)
Preparing the private DPO environment; this may take a few minutes.
{'transformers': '4.51.3', 'trl': '0.18.2', 'datasets': '3.5.0', 'accelerate': '1.6.0', 'peft': '0.15.2', 'huggingface-hub': '0.36.2', 'tokenizers': '0.21.4'}
PyTorch: 2.11.0+cu128
Transformers loaded from: /kaggle/working/.dpo_env/lib/python3.13/site-packages/transformers/__init__.py
DPO setup checks passed. Continue to the next cell.
Installation log: /kaggle/working/dpo_setup.log
import json
import math
import random
from pathlib import Path
import torch
from datasets import Dataset, load_dataset
from transformers import AutoModelForCausalLM, AutoTokenizer, set_seed
SEED = 42
set_seed(SEED)
assert torch.cuda.is_available(), "Enable a Kaggle GPU accelerator, then start a fresh session."
USE_BF16 = torch.cuda.is_bf16_supported()
USE_FP16 = not USE_BF16
print("PyTorch:", torch.__version__)
print("GPU:", torch.cuda.get_device_name(0))
print("Visible GPUs:", torch.cuda.device_count())
print("Training mixed precision:", "bf16" if USE_BF16 else "fp16")
PyTorch: 2.11.0+cu128 GPU: Tesla T4 Visible GPUs: 1 Training mixed precision: bf16
The instruction-tuned starting model lets us focus on preference learning rather than repeat SFT. Float32 parameters with automatic mixed precision work on T4; bfloat16 is enabled only when the GPU supports it.
from trl import DPOConfig, DPOTrainer
MODEL_ID = "HuggingFaceTB/SmolLM2-135M-Instruct"
DATASET_ID = "trl-lib/ultrafeedback_binarized"
N_TRAIN = 128
N_EVAL = 16
MAX_STEPS = 32
MAX_PROMPT_LENGTH = 128
MAX_COMPLETION_LENGTH = 128
OUTPUT_DIR = Path("/kaggle/working/dpo_smol_demo")
OUTPUT_DIR.mkdir(parents=True, exist_ok=True)
tokenizer = AutoTokenizer.from_pretrained(MODEL_ID)
tokenizer.pad_token = tokenizer.eos_token
tokenizer.padding_side = "right"
model = AutoModelForCausalLM.from_pretrained(
MODEL_ID, torch_dtype=torch.float32, attn_implementation="sdpa",
).to("cuda")
model.config.pad_token_id = tokenizer.pad_token_id
model.config.use_cache = False
tokenizer_config.json: 0.00B [00:00, ?B/s]
vocab.json: 0.00B [00:00, ?B/s]
merges.txt: 0.00B [00:00, ?B/s]
tokenizer.json: 0.00B [00:00, ?B/s]
special_tokens_map.json: 0%| | 0.00/655 [00:00<?, ?B/s]
config.json: 0%| | 0.00/861 [00:00<?, ?B/s]
model.safetensors: 0%| | 0.00/269M [00:00<?, ?B/s]
generation_config.json: 0%| | 0.00/132 [00:00<?, ?B/s]
Each record has an explicit prompt and two alternative assistant completions. If a source record includes the prompt inside both alternatives, we extract their common prefix first. We keep only short, nonempty pairs so truncation does not remove the entire answer or obscure the teaching example.
We stream a small subset instead of downloading the full training set. The held-out pairs come from a disjoint part of this subset. They are a diagnostic, not a full benchmark.
def normalize_pair(example):
chosen, rejected = example["chosen"], example["rejected"]
if isinstance(chosen, str) and isinstance(rejected, str):
if not example.get("prompt"):
raise ValueError("String completions require an explicit prompt.")
return {"prompt": example["prompt"], "chosen": chosen, "rejected": rejected}
# Remove a shared conversation prefix if present in both alternatives.
common = 0
while common < min(len(chosen), len(rejected)) and chosen[common] == rejected[common]:
common += 1
prompt = chosen[:common] if common else example.get("prompt")
if not prompt:
raise ValueError("No shared prompt found in this preference pair.")
return {"prompt": prompt, "chosen": chosen[common:], "rejected": rejected[common:]}
def fits_budget(row):
if isinstance(row["prompt"], str):
prompt_text = row["prompt"]
alternatives = [row["chosen"], row["rejected"]]
else:
prompt_text = tokenizer.apply_chat_template(row["prompt"], tokenize=False, add_generation_prompt=True)
alternatives = ["\n".join(m["content"] for m in row[k]) for k in ("chosen", "rejected")]
return (bool(row["chosen"]) and bool(row["rejected"]) and row["chosen"] != row["rejected"]
and len(tokenizer.encode(prompt_text)) <= MAX_PROMPT_LENGTH
and all(0 < len(tokenizer.encode(t)) <= MAX_COMPLETION_LENGTH - 16 for t in alternatives))
rows = []
stream = load_dataset(DATASET_ID, split="train", streaming=True)
for index, example in enumerate(stream):
row = normalize_pair(example)
if fits_budget(row):
rows.append(row)
if len(rows) >= N_TRAIN + N_EVAL or index >= 19999:
break
assert len(rows) >= N_TRAIN + N_EVAL, "Not enough short pairs; increase the scan limit or length budget."
random.Random(SEED).shuffle(rows)
train_dataset = Dataset.from_list(rows[:N_TRAIN])
eval_dataset = Dataset.from_list(rows[N_TRAIN:N_TRAIN + N_EVAL])
print(train_dataset)
print(json.dumps(train_dataset[0], indent=2, ensure_ascii=False))
README.md: 0%| | 0.00/643 [00:00<?, ?B/s]
Dataset({
features: ['prompt', 'chosen', 'rejected'],
num_rows: 128
})
{
"prompt": [
{
"content": "Q: In this task your given two statements in Swahili. You must judge whether the second sentence is the cause or effect of the first one. Label the instances as \"cause\" or \"effect\" based on your judgment. The sentences are separated by a newline character.\nWahamiaji walikamatwa wakikaa nchini kinyume na sheria.\nWaliondolewa kutoka nchi.\nA:",
"role": "user"
}
],
"chosen": [
{
"content": "Effect\nConfidence: 90%",
"role": "assistant"
}
],
"rejected": [
{
"content": "Effect. The second sentence is an effect of the first one, as it explains the reason why the person is now working in Kenya.",
"role": "assistant"
}
]
}
For a prompt $x$, preferred completion $y^+$, and rejected completion $y^-$:
$$\Delta = [\log\pi_\theta(y^+|x)-\log\pi_\theta(y^-|x)] -[\log\pi_{ref}(y^+|x)-\log\pi_{ref}(y^-|x)]$$$$L_{DPO}=-\log\sigma(\beta\Delta).$$Only completion tokens contribute to the sequence log-probabilities. The reference stays frozen. The objective increases the preference margin relative to the reference; it does not require both absolute response probabilities to move in opposite directions. The next cell uses illustrative log-probabilities, not measured training results.
import torch.nn.functional as F
beta = 0.1
for margin in [0.0, 2.0, -2.0]:
loss = -F.logsigmoid(torch.tensor(beta * margin))
print(f"Reference-adjusted preference margin={margin:+.1f}; loss={loss.item():.4f}")
Reference-adjusted preference margin=+0.0; loss=0.6931 Reference-adjusted preference margin=+2.0; loss=0.5981 Reference-adjusted preference margin=-2.0; loss=0.7981
@torch.inference_mode()
def generate_response(model, tokenizer, messages, max_new_tokens=96):
model.eval()
text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tokenizer(text, return_tensors="pt").to(next(model.parameters()).device)
with torch.autocast("cuda", dtype=torch.bfloat16 if USE_BF16 else torch.float16):
output = model.generate(
**inputs, max_new_tokens=max_new_tokens, do_sample=False,
pad_token_id=tokenizer.pad_token_id, eos_token_id=tokenizer.eos_token_id,
use_cache=True,
)
# Decode only the newly generated tokens, not the prompt.
return tokenizer.decode(output[0, inputs["input_ids"].shape[1]:], skip_special_tokens=True)
demo_prompt = [{"role": "user", "content": "Explain why the sky looks blue in two short sentences."}]
before_response = generate_response(model, tokenizer, demo_prompt)
print("Before DPO:\n", before_response)
Before DPO: The sky appears blue in two short sentences because the Earth's atmosphere scatters sunlight in all directions, including blue light, which is scattered more than other colors. This scattering effect is known as Rayleigh scattering, named after the British physicist Lord Rayleigh, who first described it in the 19th century.
The trainer handles chat formatting, completion masking, the frozen reference, and the sigmoid DPO loss. We use 128 short training pairs and 32 optimizer steps, with one pair per GPU microbatch and four accumulation steps. beta controls reference regularization; its effect is not simply a sampling temperature.
Watch loss and rewards/margins. The reported DPO rewards are implicit log-ratio scores, not scores from a separately trained reward model. These small runs may overfit and need not improve general answer quality.
training_args = DPOConfig(
output_dir=str(OUTPUT_DIR),
per_device_train_batch_size=1, per_device_eval_batch_size=1,
gradient_accumulation_steps=4,
gradient_checkpointing=True,
gradient_checkpointing_kwargs={"use_reentrant": False},
learning_rate=5e-6, lr_scheduler_type="cosine", warmup_steps=2,
max_steps=MAX_STEPS,
bf16=USE_BF16, fp16=USE_FP16,
optim="adamw_torch", max_grad_norm=1.0,
beta=0.1, loss_type="sigmoid",
max_prompt_length=MAX_PROMPT_LENGTH,
max_completion_length=MAX_COMPLETION_LENGTH,
max_length=MAX_PROMPT_LENGTH + MAX_COMPLETION_LENGTH,
eval_strategy="no", save_strategy="no", logging_steps=1,
report_to="none", push_to_hub=False, seed=SEED,
dataloader_num_workers=0,
)
trainer = DPOTrainer(
model=model, ref_model=None, args=training_args,
train_dataset=train_dataset, eval_dataset=eval_dataset,
processing_class=tokenizer,
)
assert trainer.args.n_gpu == 1, "This teaching configuration expects one visible GPU."
train_result = trainer.train()
assert math.isfinite(train_result.training_loss), "Training returned a non-finite loss."
metrics = trainer.evaluate()
print(metrics)
Extracting prompt in train dataset: 0%| | 0/128 [00:00<?, ? examples/s]
Applying chat template to train dataset: 0%| | 0/128 [00:00<?, ? examples/s]
Tokenizing train dataset: 0%| | 0/128 [00:00<?, ? examples/s]
Extracting prompt in eval dataset: 0%| | 0/16 [00:00<?, ? examples/s]
Applying chat template to eval dataset: 0%| | 0/16 [00:00<?, ? examples/s]
Tokenizing eval dataset: 0%| | 0/16 [00:00<?, ? examples/s]
| Step | Training Loss |
|---|---|
| 1 | 0.693100 |
| 2 | 0.693100 |
| 3 | 0.699600 |
| 4 | 0.698200 |
| 5 | 0.686300 |
| 6 | 0.698500 |
| 7 | 0.682300 |
| 8 | 0.690100 |
| 9 | 0.684600 |
| 10 | 0.676400 |
| 11 | 0.712300 |
| 12 | 0.677600 |
| 13 | 0.673100 |
| 14 | 0.692900 |
| 15 | 0.710300 |
| 16 | 0.684500 |
| 17 | 0.687000 |
| 18 | 0.693900 |
| 19 | 0.697700 |
| 20 | 0.692700 |
| 21 | 0.711000 |
| 22 | 0.680300 |
| 23 | 0.693200 |
| 24 | 0.658000 |
| 25 | 0.685800 |
| 26 | 0.698400 |
| 27 | 0.712500 |
| 28 | 0.697300 |
| 29 | 0.703600 |
| 30 | 0.709000 |
| 31 | 0.681400 |
| 32 | 0.681800 |
{'eval_loss': 0.6949271559715271, 'eval_runtime': 1.81, 'eval_samples_per_second': 8.84, 'eval_steps_per_second': 8.84, 'eval_rewards/chosen': -0.01863883249461651, 'eval_rewards/rejected': -0.01583346165716648, 'eval_rewards/accuracies': 0.5625, 'eval_rewards/margins': -0.002805371070280671, 'eval_logps/chosen': -61.15909957885742, 'eval_logps/rejected': -68.47699737548828, 'eval_logits/chosen': 4.027617931365967, 'eval_logits/rejected': 4.986034870147705, 'epoch': 1.0}
prompts = [
"Explain why the sky looks blue in two short sentences.",
"What is machine learning? Explain it to a ten-year-old.",
"Give three practical tips for learning Python.",
"Explain the difference between weather and climate.",
"Write a polite email declining a meeting invitation.",
"Why is sleep important? Answer briefly.",
"Describe a cat without using the word 'cat'.",
"Explain photosynthesis in simple terms.",
]
before_model = trainer.ref_model
after_model = trainer.model
assert before_model is not None, "The frozen reference model is unavailable."
before_model.eval()
after_model.gradient_checkpointing_disable()
after_model.eval()
results = []
for i, question in enumerate(prompts, 1):
messages = [{"role": "user", "content": question}]
before = generate_response(before_model, tokenizer, messages)
after = generate_response(after_model, tokenizer, messages)
changed = before != after
results.append({
"prompt": question,
"before": before,
"after": after,
"changed": changed,
})
print(f"\n{'=' * 70}")
print(f"Example {i} | Changed: {changed}")
print("PROMPT:", question)
print("\nBEFORE:\n", before)
print("\nAFTER:\n", after)
print(f"\nDifferent responses: {sum(r['changed'] for r in results)}/{len(results)}")
(OUTPUT_DIR / "multiple_response_comparison.json").write_text(
json.dumps(results, indent=2, ensure_ascii=False)
)
====================================================================== Example 1 | Changed: False PROMPT: Explain why the sky looks blue in two short sentences. BEFORE: The sky appears blue in two short sentences because the Earth's atmosphere scatters sunlight in all directions, including blue light, which is scattered more than other colors. This scattering effect is known as Rayleigh scattering, named after the British physicist Lord Rayleigh, who first described it in the 19th century. AFTER: The sky appears blue in two short sentences because the Earth's atmosphere scatters sunlight in all directions, including blue light, which is scattered more than other colors. This scattering effect is known as Rayleigh scattering, named after the British physicist Lord Rayleigh, who first described it in the 19th century. ====================================================================== Example 2 | Changed: False PROMPT: What is machine learning? Explain it to a ten-year-old. BEFORE: Machine learning is like a special kind of magic that helps computers learn and make decisions just like humans do. Imagine you have a big box of toys, and you want to know how to play with them. You can ask your teacher for help, and they will teach you how to play with the toys. Machine learning is like that magic box, but instead of toys, it's computers. Machine learning is a way for computers to learn from the things they see and learn AFTER: Machine learning is like a special kind of magic that helps computers learn and make decisions just like humans do. Imagine you have a big box of toys, and you want to know how to play with them. You can ask your teacher for help, and they will teach you how to play with the toys. Machine learning is like that magic box, but instead of toys, it's computers. Machine learning is a way for computers to learn from the things they see and learn ====================================================================== Example 3 | Changed: True PROMPT: Give three practical tips for learning Python. BEFORE: Here are three practical tips for learning Python: 1. **Use the `itertools` module**: The `itertools` module provides a `zip_longest` function that can efficiently generate tuples from iterables. This is a great way to iterate over lists, tuples, and other iterables in Python. 2. **Use the `re` module**: The `re` module provides a `findall` function that can find all occurrences of a pattern in a AFTER: Here are three practical tips for learning Python: 1. **Use the `itertools` module**: The `itertools` module provides a `zip_longest` function that can efficiently combine multiple iterables into a single list. This is particularly useful when working with lists of different lengths. 2. **Use the `re` module**: The `re` module provides a `findall` function that can find all occurrences of a pattern in a string. This is ====================================================================== Example 4 | Changed: True PROMPT: Explain the difference between weather and climate. BEFORE: Weather and climate are two closely related but distinct concepts in the field of meteorology. Weather refers to the short-term atmospheric conditions that occur at a particular location at a particular time. It is the state of the air, including temperature, humidity, wind, and precipitation. Weather is often described as a "what" and "when" in a given area. For example, a sunny day in New York City might be described as "it's sunny" and " AFTER: Weather and climate are two closely related but distinct concepts in the field of meteorology. Weather refers to the short-term atmospheric conditions that occur within a specific geographic area, such as temperature, humidity, wind, and precipitation. Weather is a snapshot of the atmosphere at a particular moment, and it can change quickly due to factors like wind, temperature, and solar radiation. Weather can be influenced by various factors, including the position of the sun, the time of day, ====================================================================== Example 5 | Changed: True PROMPT: Write a polite email declining a meeting invitation. BEFORE: Subject: Request for a Meeting Dear Hugging Face, I hope this email finds you well. I am writing to request a meeting with you to discuss a project I am working on. I understand that you are busy, but I would like to propose a meeting to discuss the project in more detail. I am currently working on a project that involves a significant amount of data and involves a lot of collaboration with other teams. I am hoping to get your AFTER: Subject: Request for a Meeting Dear Hugging Face, I hope this email finds you well. I am writing to request a meeting with you to discuss a project I am working on. I understand that you are busy, but I would like to propose a meeting to discuss the project further. I am currently working on a project that involves a significant amount of data and involves a lot of collaboration. I am hoping to get your input and insights to help ====================================================================== Example 6 | Changed: True PROMPT: Why is sleep important? Answer briefly. BEFORE: Sleep is important for several reasons. It helps regulate your body's internal clock, which is essential for maintaining a healthy sleep-wake cycle. It also plays a crucial role in memory consolidation, allowing you to retain information and learn new skills. Additionally, sleep helps your body repair and rejuvenate, reducing the risk of chronic diseases such as heart disease, diabetes, and obesity. In essence, getting enough sleep is vital for overall health and well-being. AFTER: Sleep is important for several reasons. It helps regulate your body's internal clock, which is essential for maintaining a healthy sleep-wake cycle. It also plays a crucial role in memory consolidation, allowing you to retain information and learn new skills. Additionally, sleep helps your body repair and rejuvenate, reducing the risk of chronic diseases such as heart disease, diabetes, and obesity. In a healthy sleep environment, you should aim to get 7-9 hours of sleep each ====================================================================== Example 7 | Changed: False PROMPT: Describe a cat without using the word 'cat'. BEFORE: A cat without using the word 'cat' is a playful and energetic feline that loves to play and explore its surroundings. AFTER: A cat without using the word 'cat' is a playful and energetic feline that loves to play and explore its surroundings. ====================================================================== Example 8 | Changed: True PROMPT: Explain photosynthesis in simple terms. BEFORE: Photosynthesis is a process that plants, algae, and some bacteria use to make their own food. It's a really cool way for plants to get energy from the sun. Imagine you have a big tree that's got a lot of leaves. When the sun shines on the leaves, it makes a special thing called sunlight. This sunlight is made up of tiny things called light energy. Plants use a special part of their leaves called chloroplasts to make this sunlight AFTER: Photosynthesis is a process used by plants, algae, and some bacteria to convert light energy from the sun into chemical energy stored in glucose. Here's a simplified explanation: Plants use sunlight to convert carbon dioxide and water into glucose and oxygen. This process is called photosynthesis. Here's what happens: 1. **Light Absorption**: Plants absorb light energy from the sun. This energy is captured by chlorophyll, a green pigment in the leaves. 2. Different responses: 5/8
7361
Adapted from Hugging Face Smol Course, Preference Alignment with Direct Preference Optimization (DPO), by the Hugging Face team and Smol Course contributors. Original notebook: https://github.com/huggingface/smol-course/blob/main/v1/2_preference_alignment/notebooks/dpo_finetuning_example.ipynb . The source project is licensed under Apache 2.0: https://github.com/huggingface/smol-course/blob/main/LICENSE .
Method: Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning, and Chelsea Finn, Direct Preference Optimization: Your Language Model is Secretly a Reward Model (2023): https://arxiv.org/abs/2305.18290 .
Implementation reference (version used here): https://huggingface.co/docs/trl/v0.18.2/dpo_trainer . Model: HuggingFaceTB/SmolLM2-135M-Instruct. Preference data: trl-lib/ultrafeedback_binarized, derived from UltraFeedback by Cui et al.
This revision changes environment setup, GPU precision, data size, diagnostics, and local saving for a compact Kaggle teaching run. It does not reuse previously executed training results.