4.3 RLHF & Alignment (PPO, GRPO, DPO)
Interview-ready field notes
The 20-second answer
Alignment training makes a capable base model more helpful and safer by learning which responses people prefer, then optimizing toward those preferences with constraints.
Core ideas to retain
- A common pipeline is SFT, preference data, then preference optimization. Preference pairs say which of two responses humans prefer.
- PPO uses a reward signal with clipped policy updates; a KL penalty keeps the policy near a reference model.
- DPO optimizes preference pairs directly without fitting an explicit reward model or running an online RL loop.
- GRPO is a group-relative policy optimization approach: compare multiple sampled answers for the same prompt to form an advantage baseline.
Interview / OA rule
Do not claim alignment makes a model truthful or safe by itself. It optimizes measured preferences; reward misspecification, data gaps, and distribution shift remain.
One good written resource
Direct Preference Optimization paper — read this after the video when you want a clearer mental model, not more pages of notes.
Most asked
Interview questions to practise aloud
Each answer is the level of detail expected for a strong fundamentals round.
What problem does RLHF solve?
Pretraining teaches prediction, not necessarily preferred assistant behavior. RLHF uses human preference signals to steer helpfulness, style, and safety behavior.
PPO vs DPO?
PPO optimizes a reward through an RL policy update and usually uses a reference/KL constraint. DPO directly trains on chosen-versus-rejected pairs with a derived objective.
Why use a KL penalty in RLHF?
It limits how far the policy drifts from a capable reference, helping prevent reward hacking and loss of language quality.