Deep systems understanding · Study map 60 of 85
Separate prefill and decode: two pools, one conversation
Suggested preparation: Chunked prefill: stop one reader blocking every writerInterconnects: the slowest road can set the pace
Picture it
Reading a thick textbook requires deep, continuous focus, while answering rapid-fire quiz questions demands quick, rhythmic reflexes. A single worker trying to switch between digesting massive new prompts and dripping out single tokens will inevitably stutter. By splitting these tasks into two separate, specialized rooms—one built for heavy reading and one for rapid typing—the system can maintain a fluid, unbroken stream of answers, provided they smoothly hand off their notes in between.
What to understand
- Independent pool sizing, TP layouts, KV transfer, and handoff
- Aggregated versus disaggregated serving; goodput under mixed demand
Follow the references
1 · Paper
Zhong et al. · DistServe ↗Read placement and bandwidth assumptions before the goodput plots.
2 · Documentation
NVIDIA · Dynamo ↗Follow disaggregated serving and the KV-transfer path for your backend.
Keep the distinction. Disaggregation is a measured deployment choice, not a universal upgrade. Version-specific connector and backend support must be checked.
Make it concrete
Compare colocated chunked prefill with separate pools at the same total GPU count. Sweep prompt/output lengths and load; include transfer time, idle capacity, TTFT, and ITL.
You understand it when…
Why might separation lose on short prompts, low traffic, or slow links?
This is a study map. Follow the reference for the full explanation; use the question to test your mental model. Browse the reference shelf.