Broad intuition · Study map 15 of 85
Optimizers: walking with noisy directions
Suggested preparation: Backpropagation: trace influence backward
Picture it
The gradient points locally uphill; learning steps usually go the other way. Mini-batches provide noisy directions. Momentum remembers recent motion, while Adam also rescales coordinates using running gradient statistics.
What to understand
- Learning rate, batch size, momentum, Adam, and schedules
- Vanishing/exploding gradients, initialization, clipping, and numerical stability
Follow the references
You understand it when…
Why can a larger learning rate initially look faster and then make the loss diverge?
This is a study map. Follow the reference for the full explanation; use the question to test your mental model. Browse the reference shelf.