Broad intuition · Study map 12 of 85
Evaluation: the scoreboard shapes the system
Suggested preparation: Generalization: learning the pattern, not the answer sheet
Picture it
One average score can hide the failures that matter. A medical detector, search engine, and chatbot need different scoreboards. Test on the users and situations the system will actually face, including rare and difficult cases.
What to understand
- Precision/recall, thresholds, PR and ROC curves, calibration
- Temporal/group splits, uncertainty intervals, slice metrics, online experiments
Follow the references
1 · Course
Google · Machine Learning Crash Course ↗Use classification metrics and fairness sections. Move a threshold and describe who benefits and who loses.
2 · Code + guide
EleutherAI · Language Model Evaluation Harness ↗Inspect task configuration and result output; identify the prompt and scoring assumptions.
Keep the distinction. Repeated tuning on a benchmark gradually turns it into training data in practice.
You understand it when…
Why can 99% accuracy be useless when only 1% of cases are positive?
This is a study map. Follow the reference for the full explanation; use the question to test your mental model. Browse the reference shelf.