Distributed training: synchronize learning

Picture it

Think of painting a massive mural. You can have many painters work on different sections and blend their colors at the seams, or you can split a single brushstroke across multiple arms. Each approach requires coordination. If one painter is slow, or if the paint buckets are too far apart, the whole masterpiece stalls.

What to understand

• DDP, all-reduce, FSDP/ZeRO stages, tensor and pipeline parallelism
• Global batch, accumulation, synchronization, and communication overlap

Follow the references

1. 1 · Paper

   Rajbhandari et al. · ZeRO ↗

   Follow how optimizer, gradient, and parameter partitioning progressively reduce replicated state.

2. 2 · Documentation

   NVIDIA · NCCL collectives ↗

   Map each training communication step to a collective.

Make it concrete

Draw parameter/gradient/optimizer placement for DDP and fully sharded training. Estimate memory per device and identify the collective required at each stage.