Continuous batching: refill seats every iteration

Picture it

Picture a Ferris wheel where the operator waits for every single rider to finish a 10-minute ride before letting anyone new get on. Continuous batching changes the rules, allowing new riders to hop into empty carts the moment someone steps off, keeping the wheel constantly packed. This beautifully orchestrates the GPU's momentum, ensuring no seat is ever empty, though it demands a brilliant juggler to keep track of everyone's tickets.

What to understand

• Iteration-level scheduling, admission, and token budgets
• Batch size versus latency, memory, fairness, and throughput

Follow the references

1. 1 · Book chapter

   How To Scale Your Model · Transformer inference ↗

   Read continuous batching and compare it with a fixed request batch.

2. 2 · Documentation

   vLLM · Optimization and tuning ↗

   Study the sequence and token-budget controls rather than copying default values.

Make it concrete

Sweep concurrency on a mixed-length workload. Find the point where throughput gains flatten while p95 latency or preemption rises.