Deep systems understanding · Study map 82 of 85
Lab 2 · Keep chats smooth during document bursts
Suggested preparation: Lab 1 · Explain a single GPU’s limits
Picture it
Picture a highway shared by swift motorcycles and massive cargo trucks. If you let the trucks monopolize the lanes, the motorcycles get stuck in a frustrating traffic jam. You need a scheduling policy that weaves them together, ensuring the heavy loads reach their destination without forcing the fast, lightweight traffic to a miserable crawl.
What to understand
- Continuous batching, chunk size, caching, and admission
- Per-class fairness and equal-resource comparisons
Follow the references
1 · Paper
Agrawal et al. · SARATHI ↗Use its scheduling idea to form your hypothesis.
2 · Documentation
vLLM · Optimization and tuning ↗Map that hypothesis to the current engine’s supported controls.
Make it concrete
Compare a baseline with chunking and prefix caching on realistic repeated/unique prefixes. Report per-class TTFT, ITL, rejection rate, goodput, memory, and total GPU count.
You understand it when…
What is the worst-served class after your optimization, and is it still within its target?
This is a study map. Follow the reference for the full explanation; use the question to test your mental model. Browse the reference shelf.