Deep systems understanding · Study map 70 of 85
Streaming: deliver tokens without losing control
Suggested preparation: Multimodal serving: text tokens miss part of the bill
Picture it
Generating tokens is like pouring a steady stream of water from a pitcher. If the model pours too fast and the user's connection is a tiny, slow funnel, the water will violently splash back and flood the server's internal buffers. A graceful streaming gateway monitors this flow in real time, pulling back the pitcher when the client struggles, and cleanly shutting off the tap the moment the user walks away, ensuring not a single drop of expensive GPU computation is wasted.
What to understand
- SSE/WebSocket behavior, proxy buffering, and time-to-visible-token
- Connection limits, cancellation, queue bounds, and stream errors
Follow the references
1 · Documentation
vLLM · Production metrics ↗Compare engine latency with client-observed latency and end-to-end request timing.
2 · Book chapter
Google SRE · Handling overload ↗Apply bounded-resource reasoning to connections and output buffers.
Make it concrete
Test through the real gateway with a slow reader and a disconnect. Verify prompt cancellation releases cache and active-request slots; compare client and server timestamps.
You understand it when…
Where would you look if the engine’s ITL is low but the browser receives text in large bursts?
This is a study map. Follow the reference for the full explanation; use the question to test your mental model. Browse the reference shelf.