Deep systems understanding · Study map 35 of 85
Prefill and decode: reading versus writing
Suggested preparation: The life of an inference request
Picture it
Imagine reading a thick dossier in one sitting to understand the context—this bulk processing is the 'prefill' phase, which builds a mental summary and produces your first response. Then, you enter the 'decode' phase, where you utter one word at a time, glancing back at your summary before speaking the next word. Prefill is a deep, compute-heavy reading marathon, while decode is a rapid, memory-intensive stutter of fetching the right context for each new syllable.
What to understand
- Parallel prompt processing versus sequential generation
- Compute, HBM bandwidth, launch overhead, and network bottlenecks
Follow the references
1 · Book chapter
How To Scale Your Model · Transformer inference ↗Compare the linear-operation and attention bottleneck sections. Notice how batch size changes the answer.
2 · Paper
Agrawal et al. · SARATHI ↗Read the motivation for mixing decode work with chunks of prefill.
Keep the distinction. These are common regimes, not laws. Large decode batches can become compute-bound; dense prefill attention is quadratic in prompt length, but the whole transformer is not only attention.
Make it concrete
Compare short/long prompts with short/long outputs at equal concurrency. Record TTFT and inter-token latency separately.
You understand it when…
Why can inserting one very long prompt make existing users’ token streams stutter?
This is a study map. Follow the reference for the full explanation; use the question to test your mental model. Browse the reference shelf.