Preferences, rewards, and alignment

Picture it

Demonstrations teach what a good answer looks like; preferences teach which of two answers is better. RLHF can optimize a learned reward. DPO uses preference pairs directly to change relative response probabilities. Both inherit weaknesses in their feedback.

What to understand

• Reward models, reference policies, PPO, DPO, and verifiable rewards
• Reward hacking, distribution shift, safety evaluation

Follow the references

1. 1 · Paper

   Rafailov et al. · Direct Preference Optimization ↗

   Read the problem setup and contrast its preference loss with the RLHF pipeline.

2. 2 · Course

   Hugging Face · Deep reinforcement learning course ↗

   Review policy and reward before connecting reinforcement learning to language generation.