1.3 Probability & Information Theory
Interview-ready field notes
The 20-second answer
Probability models uncertainty; information theory measures how surprised we are. Cross-entropy turns wrong or low-confidence predictions into a training loss.
Core ideas to retain
- Bayes' rule updates a prior using evidence: posterior likelihood prior.
- Expectation is the average under a distribution; variance measures spread around that average.
- Entropy is uncertainty. A uniform distribution has high entropy; a certain outcome has zero entropy.
- Cross-entropy is minimized when predicted matches target . For a one-hot label it is .
Interview / OA rule
Connect losses to likelihoods: Gaussian noise gives MSE; categorical or Bernoulli likelihood gives cross-entropy. Never call KL divergence a distance: it is asymmetric.
One good written resource
Seeing Theory — Probability — read this after the video when you want a clearer mental model, not more pages of notes.
Most asked
Interview questions to practise aloud
Each answer is the level of detail expected for a strong fundamentals round.
Why is cross-entropy used for classification?
It is the negative log-likelihood of the true class. It heavily penalizes assigning a tiny probability to the correct answer and gives useful gradients.
Entropy vs cross-entropy vs KL?
Entropy is uncertainty in one distribution; cross-entropy scores a predicted distribution against the true one; KL is the extra cross-entropy beyond true entropy.
What is MLE?
Choose parameters that make observed data most probable. We usually minimize negative log-likelihood because sums are easier than products.