4.5 Quantization & Efficiency
Interview-ready field notes
The 20-second answer
Quantization stores or computes model values with fewer bits, cutting memory and often speeding inference while accepting some numerical error.
Core ideas to retain
- FP32, BF16/FP16, INT8, and 4-bit formats trade precision for memory. Weight memory is roughly parameters bytes per value.
- Post-training quantization converts an existing model; quantization-aware training simulates quantization while training and can preserve accuracy better.
- Per-channel or group-wise scales usually preserve quality better than one global scale because tensor ranges differ.
- Quantizing weights reduces model memory; long-context generation can still be dominated by KV-cache memory and bandwidth.
Interview / OA rule
Say what is quantized: weights, activations, KV cache, or all three. Memory savings do not automatically mean proportional speedup; hardware and bandwidth matter.
One good written resource
Hugging Face — Quantization — read this after the video when you want a clearer mental model, not more pages of notes.
Most asked
Interview questions to practise aloud
Each answer is the level of detail expected for a strong fundamentals round.
Why quantize an LLM?
To fit it in less memory and often improve throughput or latency. The cost is approximation error and possible hardware/kernel constraints.
INT8 vs 4-bit?
Four-bit saves more memory but introduces more error and may require better quantization schemes. INT8 is generally a safer quality–efficiency compromise.
What is QLoRA?
It stores a frozen base model in low-bit form and trains LoRA adapters, enabling fine-tuning with much lower memory than full precision full fine-tuning.