Deep systems understanding · Study map 34 of 85
The life of an inference request
Suggested preparation: Tokens: the model’s units of workAttention: each token asks what matters hereSampling, reasoning, and inference-time compute
Picture it
Think of a user's request as a droplet of water flowing through a sprawling aqueduct system. It must navigate intake valves, holding tanks, and narrow pipelines before finally splashing into the high-speed turbine of the GPU. Even if the turbine spins blazingly fast, a clogged pipe upstream will slow the entire flow to a trickle. To fix a slow stream, trace the droplet's entire journey before simply upgrading the turbine.
What to understand
- Gateway → tokenizer → scheduler → model → sampler → stream
- Model/tokenizer revisions, chat templates, cancellation, and error handling
Follow the references
1 · Documentation
vLLM · Production metrics ↗Find queue, prefill, decode, and end-to-end latency metrics. Map each to a request stage.
2 · Documentation
SGLang · Serving documentation ↗Follow the quickstart request path and identify the engine versus HTTP-server responsibilities.
Make it concrete
Draw a request timeline. Run or inspect a small-model server trace; label CPU time, waiting, prefill, and decode. Keep this baseline for later experiments.
You understand it when…
The first token is slow but later tokens are fast. Name three places to investigate before changing hardware.
This is a study map. Follow the reference for the full explanation; use the question to test your mental model. Browse the reference shelf.