2.4 Classical ML Algorithms
Interview-ready field notes
The 20-second answer
Classical ML chooses different inductive biases: linear models favor simple boundaries, trees favor rules, neighbors favor local similarity, and SVMs favor wide margins.
Core ideas to retain
- Linear regression predicts a number; logistic regression applies sigmoid to model a class probability and still has a linear boundary in feature space.
- A decision tree greedily splits features. A random forest bags many decorrelated trees to reduce variance.
- Gradient boosting trains small trees sequentially to correct earlier errors; it mainly reduces bias but needs regularization.
- k-NN is a lazy distance-based method; SVM maximizes margin and can use kernels for nonlinear boundaries.
Interview / OA rule
The classic contrast: bagging trains models independently and averages to reduce variance; boosting trains sequentially to reduce bias.
One good written resource
scikit-learn — Supervised Learning Guide — read this after the video when you want a clearer mental model, not more pages of notes.
Most asked
Interview questions to practise aloud
Each answer is the level of detail expected for a strong fundamentals round.
Random forest vs gradient boosting?
A random forest trains trees independently on resampled data and averages them; boosting trains trees one after another to correct residual errors.
Why does a decision tree overfit?
It can keep making narrow splits that memorize noise. Limit depth or leaf size, prune it, or use an ensemble.
When is k-NN a poor choice?
For high-dimensional, very large, or poorly scaled data. Distances become less informative and inference requires searching stored examples.