Deep systems understanding · Study map 36 of 85
The scoreboard: latency, throughput, and goodput
Suggested preparation: Prefill and decode: reading versus writing
Picture it
Picture a bustling pizza kitchen with a massive oven. Throughput is the sheer number of pizzas sliding out the door every hour, while latency is how long one specific customer taps their foot waiting for dinner. Goodput is the true measure of success: the number of perfectly baked pizzas that actually reach tables before the customers get impatient and leave. Balancing the kitchen isn't just about maximizing the oven's fire, but ensuring a smooth, timely flow from dough to dining table.
What to understand
- TTFT, inter-token latency, time per output token, end-to-end latency
- p50/p95/p99, request versus token throughput, SLOs, and goodput
Follow the references
1 · Paper
Zhong et al. · DistServe ↗Read how the evaluation combines first-token and per-token latency constraints.
2 · Documentation
vLLM · Serving benchmark reference ↗Inspect supported latency metrics and goodput/SLO options for the installed version.
Keep the distinction. Average time per output token can hide pauses between individual tokens. State exactly how every metric is computed.
Make it concrete
Write separate SLOs for interactive chat and offline extraction. Plot throughput against p95 TTFT as offered load rises.
You understand it when…
A configuration doubles tokens/second but misses the p95 latency target. When is it a regression?
This is a study map. Follow the reference for the full explanation; use the question to test your mental model. Browse the reference shelf.