Papers that explain the mechanisms
First read the problem, system diagram, assumptions, and evaluation setup. Derivations can come later. Published speedups describe the paper’s workload and baseline, not a promise for yours.
The GPU keeps visiting a giant scratchpad
Find which HBM reads and writes the algorithm removes; distinguish arithmetic complexity from memory traffic.
Paper · Dao et al. · FlashAttention ↗
Growing conversations waste memory
Trace a block table and two branching sequences before looking at throughput plots.
Long prompts interrupt streams
Compare interleaving small prefills with separating prefill and decode. Inspect transfer and scheduling assumptions.
- Chunked prefill: stop one reader blocking every writer →
- Separate prefill and decode: two pools, one conversation →
Paper · Agrawal et al. · SARATHI ↗
Paper · Zhong et al. · DistServe ↗
An expensive model generates one step at a time
Trace a draft, its verification, and a rejected token. Explain the correction before using the speedup chart.
Capacity and active work are different
Compare low-rank updates, sparse experts, and latent attention caches. Identify what each saves and what remains resident.
- LoRA: learn a small correction →
- Mixture of experts: sparse work, uneven traffic →
- KV precision and architectural compression →
Paper · Hu et al. · LoRA ↗
Paper · Jiang et al. · Mixtral of Experts ↗
Paper · DeepSeek · DeepSeek-V2 ↗
Learning and memory can be organized differently
Contrast preference optimization, recurrent state, and sharded training state.
- Preferences, rewards, and alignment →
- Sequences: recurrence, attention, and state →
- Distributed training: synchronize learning →
Paper · Rafailov et al. · Direct Preference Optimization ↗
Paper · Gu & Dao · Mamba ↗
Paper · Rajbhandari et al. · ZeRO ↗