Chunked prefill: stop one reader blocking every writer

Picture it

Imagine a crowded post office where someone walks in with a thousand packages, forcing the clerk to process them all while a line of people holding single letters waits in agony. Chunked prefill solves this by politely asking the heavy shipper to process their packages in batches of ten, allowing the clerk to rapidly stamp single letters in between. This keeps the line moving smoothly for everyone, even if it means the giant shipment takes a little longer to finish.

What to understand

• Prefill/decode interference and token budgets
• Chunk size, scheduling policy, fairness, and TTFT/ITL tradeoffs

Follow the references

1. 1 · Paper

   Agrawal et al. · SARATHI ↗

   Follow the mixed-batch diagrams and the role of chunk size.

2. 2 · Documentation

   vLLM · Optimization and tuning ↗

   Read chunked prefill guidance and measure the current scheduler’s behavior.

Make it concrete

Replay short chats plus long documents. Sweep the token budget and compare p95 TTFT, inter-token gaps, throughput, and starvation of long requests.