Broad intuition · Study map 24 of 85
Pretraining, instruction tuning, and distillation
Suggested preparation: Attention: each token asks what matters here
Picture it
Pretraining learns broad patterns from lots of data. Instruction tuning demonstrates how to respond. Distillation teaches a smaller student using a stronger teacher’s behavior. These change the weights; putting examples in a prompt usually does not.
What to understand
- Next-token training, data quality, deduplication, scaling laws
- SFT, transfer, catastrophic forgetting, distillation, and data contamination
Follow the references
1 · Course + videos
Stanford · CS336 Language Modeling from Scratch (2025) ↗Use data, scaling laws, and supervised fine-tuning lectures. Identify the objective and dataset at each stage.
You understand it when…
If a model needs fresh private facts, when would you retrieve them rather than retrain it?
This is a study map. Follow the reference for the full explanation; use the question to test your mental model. Browse the reference shelf.