KV cache: keep the work you can reuse

Picture it

Imagine writing a novel and having to reread the entire book from page one just to figure out what the next word should be. Instead of this maddening repetition, a KV cache acts like a perfectly organized set of character dossiers and plot summaries updated with each word. When it's time to write the next sentence, the model simply consults its existing dossiers, sparing the brain the exhausting work of reading the whole book again.

What to understand

• Cache contents per layer; attention reads and cache writes
• MHA, MQA, GQA, sliding windows, and cache lifetime

Follow the references

1. 1 · Book chapter

   How To Scale Your Model · Transformer inference ↗

   Follow the cached sampling diagram and the attention bandwidth analysis.

Make it concrete

Estimate cache growth at 1k, 8k, and 32k tokens for MHA versus GQA. Predict the effect on capacity and decode latency.