Deep systems understanding · Study map 61 of 85
Route to cached work without creating a hotspot
Suggested preparation: Separate prefill and decode: two pools, one conversation
Picture it
Imagine a fleet of taxis where some drivers have already navigated to your specific neighborhood and know the shortcuts. If you blindly assign the next customer to the least busy driver, they might waste time learning the route from scratch. Cache-aware routing acts as a smart dispatcher, sending requests to the exact worker who already holds the context in memory, perfectly balancing the desire to reuse past work with the need to prevent any single driver from getting overwhelmed.
What to understand
- Least-loaded versus token-work-aware routing
- Prefix affinity, stale cache metadata, load estimates, and topology
Follow the references
1 · Documentation
NVIDIA Dynamo · Router design ↗Follow the cache-overlap and active-load components of worker selection.
Make it concrete
Replay repeated-prefix traffic with bursts. Compare round robin, least-loaded, and cache-aware routing using cache reuse, queue time, p95 TTFT, and worker imbalance.
You understand it when…
When should a request go to a colder worker even though another worker has its entire prefix cached?
This is a study map. Follow the reference for the full explanation; use the question to test your mental model. Browse the reference shelf.