Deep systems understanding · Study map 48 of 85
Prefix caching: reuse the shared first pages
Suggested preparation: Chunked prefill: stop one reader blocking every writer
Picture it
Imagine a tour guide who recites the same lengthy historical introduction to every new group of tourists before answering their specific questions. Prefix caching is like recording that introduction on a tape; when a new group arrives, you just hit play on the shared history. The guide still has to actively think to answer the group's unique questions, but they save immense energy by not repeating the standard prologue.
What to understand
- Exact token-prefix reuse, cache keys, model/adapter identity
- Hit rate by reused tokens, eviction, privacy, and cache-aware placement
Follow the references
1 · Documentation
vLLM · Automatic prefix caching ↗Read the examples and limitations. Distinguish skipped prefill from unchanged decode work.
2 · Documentation
NVIDIA Dynamo · Router design ↗Inspect how cache overlap competes with current worker load.
Keep the distinction. A cache hit is not a reusable final answer. Isolate tenants appropriately and never use cache identity to bypass authorization.
Make it concrete
Compare cold and warm repeated prefixes, then vary one early token. Measure reused tokens and TTFT under realistic cache capacity.
You understand it when…
Why can putting a timestamp at the start of every prompt destroy cache reuse?
This is a study map. Follow the reference for the full explanation; use the question to test your mental model. Browse the reference shelf.