The memory budget: weights are only the beginning

Picture it

Think of your GPU memory as a spacious office building. The model's weights are the heavy, permanent furniture anchored in place, taking up a fixed footprint. Every new user request is an employee wheeling in their own temporary desk and files, demanding space. Just because the building looks empty when the furniture is placed doesn't mean it won't overflow once the entire workforce arrives and starts unpacking.

What to understand

• Weight bytes ≈ parameters × bytes per parameter
• KV bytes ≈ 2 × layers × KV heads × head dimension × cached tokens × bytes per element

Follow the references

1. 1 · Book chapter

   How To Scale Your Model · Transformer inference ↗

   Use the memory section to separate fixed model cost from per-sequence cache cost.

2. 2 · Documentation

   vLLM · Optimization and tuning ↗

   Find memory pressure, preemption, and concurrency controls.

Make it concrete

For 32 layers, 8 KV heads, head dimension 128, and BF16 KV: compute 128 KiB per token, then 1 GiB for 8,192 tokens per sequence. Add weights and workspace before estimating concurrency.