Deep systems understanding · Study map 51 of 85
Quantization: fewer bits, different errors
Suggested preparation: The memory budget: weights are only the beginningRooflines: count work per byte moved
Picture it
Imagine a high-resolution painting squeezed onto a lower-resolution grid. We lose some fine details, but the image takes up far less canvas space and becomes much easier to transport. Quantization works similarly by rounding exact numbers to a coarser grid, shrinking the model's footprint in memory. This lets us glide through calculations faster, provided our hardware knows how to paint quickly with these broader brushstrokes.
What to understand
- FP16/BF16, INT8/INT4, FP8, scales, groups, outliers
- Weight-only versus weight/activation quantization; calibration and kernel support
Follow the references
1 · Documentation
vLLM · Quantization ↗Use the supported-hardware table and compare weight-only with activation-aware paths.
Keep the distinction. Re-evaluate domain-specific failures, long contexts, and tool outputs. A single benchmark average is not a quality guarantee.
Make it concrete
Compare a supported quantized model with its higher-precision baseline using the same workload. Report quality, memory, TTFT, decode rate, and full service cost.
You understand it when…
Why might a smaller quantized checkpoint be slower than BF16 on a particular GPU?
This is a study map. Follow the reference for the full explanation; use the question to test your mental model. Browse the reference shelf.