4.1 Pretraining vs Fine-tuning
Interview-ready field notes
The 20-second answer
Pretraining learns broad patterns from massive general data; fine-tuning adapts that base model to a narrower behavior, domain, or task with curated examples.
Core ideas to retain
- A causal LLM is commonly pretrained by next-token prediction, so labels come from the text itself.
- Supervised fine-tuning (SFT) trains on prompt–ideal-answer pairs to make a pretrained model follow instructions or a format.
- Fine-tuning changes weights; prompting changes only the input. Start with prompting when the model already has the needed capability.
- Fine-tuning can overfit, forget previous skills, or amplify data quality problems, so evaluate both the target task and retained capabilities.
Interview / OA rule
A clean interview answer: pretraining builds general representations and is expensive; SFT specializes behavior and is much cheaper, but depends heavily on data quality and evaluation.
One good written resource
Hugging Face — LLM Course — read this after the video when you want a clearer mental model, not more pages of notes.
Most asked
Interview questions to practise aloud
Each answer is the level of detail expected for a strong fundamentals round.
Pretraining vs fine-tuning?
Pretraining learns broad language/statistical knowledge from large unlabeled corpora; fine-tuning adapts it using smaller, task-specific training examples.
Prompting vs fine-tuning?
Prompting is immediate and leaves model weights unchanged. Fine-tuning costs data and compute but can make behavior more reliable or specialized.
What is catastrophic forgetting?
After narrow fine-tuning, a model loses performance on prior capabilities. Mix data, use conservative training, or evaluate retention to manage it.