Deep systems understanding · Study map 49 of 85
When cache fills: evict, recompute, or move
Suggested preparation: Prefix caching: reuse the shared first pages
Picture it
Think of your GPU's memory as premium desktop real estate, while the system's main RAM is a distant filing cabinet. Keeping a stack of old reference papers on your desk blocks room for active work, so you eventually have to toss them or walk them to the cabinet. If you need them again, you pay the penalty of fetching them or rewriting them from scratch—a delicate balancing act between hoarding space and wasting time.
What to understand
- Active-request preemption versus reusable-prefix eviction
- CPU/remote offload, recomputation, transfer bandwidth, and cache tiers
Follow the references
1 · Documentation
vLLM · Optimization and tuning ↗Study preemption and memory-pressure guidance; identify when a request is recomputed.
2 · Documentation
NVIDIA · Dynamo ↗Find KV cache offloading and compare storage tiers with GPU residency.
Make it concrete
Compare transfer time ≈ cache bytes / effective link bandwidth with measured prefill time. Include queueing and contention, then choose a cache policy for hot and cold prefixes.
You understand it when…
When can fetching a cache cost more time than recomputing the prompt?
This is a study map. Follow the reference for the full explanation; use the question to test your mental model. Browse the reference shelf.