5.2 Algorithm Families
Interview-ready field notes
The 20-second answer
RL algorithms differ mainly by whether they learn from values, directly optimize a policy, use a learned model, and reuse old experience.
Core ideas to retain
- Value-based methods such as DQN learn and choose the best action. They fit discrete action spaces well.
- Policy-gradient methods directly optimize a parameterized policy and naturally handle continuous actions, but estimates can be noisy.
- Actor–critic methods combine a policy (actor) with a value estimate (critic) to reduce gradient variance.
- On-policy methods learn from the current policy's fresh data; off-policy methods can reuse replayed data from older behavior policies.
Interview / OA rule
A strong one-liner: DQN is value-based and off-policy; REINFORCE is policy-gradient and on-policy; PPO is on-policy actor–critic; SAC is off-policy actor–critic for continuous control.
One good written resource
OpenAI Spinning Up — Algorithm Families — read this after the video when you want a clearer mental model, not more pages of notes.
Most asked
Interview questions to practise aloud
Each answer is the level of detail expected for a strong fundamentals round.
On-policy vs off-policy?
On-policy learns from data generated by the policy being improved; off-policy can learn from data collected by other or older policies, enabling replay.
Why does DQN use a target network?
The model is learning targets that depend on its own predictions. A slowly updated target stabilizes this moving-target problem.
Why use actor–critic?
The critic estimates value to provide a lower-variance learning signal, while the actor learns which actions to take.