2.3 Optimization
Interview-ready field notes
The 20-second answer
Optimization minimizes a loss by repeatedly estimating its gradient and updating parameters. The learning rate controls how far each update moves.
Core ideas to retain
- Batch GD uses all data, SGD uses one example, and mini-batch GD is the practical GPU-friendly compromise.
- Momentum averages past gradients to reduce zig-zagging and speed movement in a consistent direction.
- Adam combines momentum with per-parameter scaling from squared gradients. AdamW decouples weight decay from that adaptive update.
- Vanishing gradients stop early layers learning; exploding gradients make updates unstable. Residual paths, good initialization, normalization, and clipping help.
Interview / OA rule
When loss diverges, suspect the learning rate first. When training is slow but stable, try a schedule, warmup, or optimizer change before changing the architecture.
One good written resource
Google — Gradient Descent — read this after the video when you want a clearer mental model, not more pages of notes.
Most asked
Interview questions to practise aloud
Each answer is the level of detail expected for a strong fundamentals round.
Why use mini-batches?
They provide noisy but useful gradient estimates while exploiting parallel hardware; full batches are expensive and single examples are often too noisy.
Adam vs SGD with momentum?
Both use momentum. Adam also adaptively scales each coordinate using squared gradients, making it easier to tune; SGD can sometimes generalize better with careful schedules.
What is gradient clipping?
Cap a gradient's value or norm before the update. It protects training, especially sequence models, from rare exploding updates.