Deep systems understanding · Study map 50 of 85
Long context: fit, attend, and remember usefully
Suggested preparation: KV cache: keep the work you can reuseThe memory budget: weights are only the beginning
Picture it
Picture giving an assistant an infinitely expansive desk to lay out thousands of pages of documents. While they physically have the space to hold all the paper, their actual ability to find the one crucial needle in that haystack is a completely different skill. Expanding a model's context window secures the massive desk, but you still must test whether the assistant's attention span can meaningfully synthesize the sprawling information.
What to understand
- Full attention versus sliding-window, sparse, and hybrid attention
- Context parallelism, RoPE extension, truncation, compaction, and long-context evaluation
Follow the references
1 · Book chapter
How To Scale Your Model · Transformer inference ↗Compare attention costs as context grows; keep prefill arithmetic and decode cache reads separate.
2 · Documentation
vLLM · Parallelism and scaling ↗Follow context-parallel guidance and supported configurations; identify where communication replaces local work.
Keep the distinction. Changing position scaling or pruning cached tokens can change model behavior. Use only supported configurations and evaluate retrieval-sensitive tasks.
Make it concrete
Sweep context length with evidence placed near the start, middle, and end. Compare full context with retrieval or compaction; report recall, quality, memory, TTFT, and ITL.
You understand it when…
Why does passing a maximum-length input test not establish good long-document answering?
This is a study map. Follow the reference for the full explanation; use the question to test your mental model. Browse the reference shelf.