Broad intuition · Study map 8 of 85
Probability and loss: how surprising was the answer?
Suggested preparation: Derivatives: a sensitivity meter
Picture it
A model spreads probability across possible answers. Cross-entropy makes a confident wrong prediction costly. Entropy describes uncertainty in a distribution; KL divergence compares distributions in a direction-sensitive way.
What to understand
- Conditional probability, expectation, variance, and Bayes’ rule
- Log likelihood, cross-entropy, entropy, perplexity, and KL divergence
Follow the references
Keep the distinction. A fluent answer or a high next-token probability is not a calibrated probability that the whole answer is true.
You understand it when…
Why is assigning 0.001 probability to the correct answer punished more than assigning 0.4?
This is a study map. Follow the reference for the full explanation; use the question to test your mental model. Browse the reference shelf.