Deep systems understanding · Study map 72 of 85
Training memory: keep enough history to learn
Suggested preparation: Offline batches, local models, and heterogeneous fleets
Picture it
Imagine a hiker mapping a trail. In inference, you only need the final map. But in training, you must carry every breadcrumb, detour, and compass reading to find your way back and correct your route. This means saving intermediate steps, gradients, and optimizer states. Every choice to recompute a step instead of saving it is a trade-off between the weight of your pack and the time spent walking.
What to understand
- Parameters, gradients, optimizer states, and saved activations
- Micro-batches, gradient accumulation, mixed precision, and activation checkpointing
Follow the references
1 · Book chapter
How To Scale Your Model · Transformer math and training ↗Read the forward/backward and memory accounting sections.
2 · Paper
Rajbhandari et al. · ZeRO ↗Separate the three kinds of state ZeRO partitions.
Make it concrete
Build a memory ledger for a small dense model. Compare full fine-tuning, LoRA, and checkpointed training; state which tensors still exist in each case.
You understand it when…
Why does an inference memory estimate badly underpredict the memory needed for Adam training?
This is a study map. Follow the reference for the full explanation; use the question to test your mental model. Browse the reference shelf.