Deep systems understanding · Study map 58 of 85
Interconnects: the slowest road can set the pace
Suggested preparation: Replicate or shard? DP, TP, PP, and CP
Picture it
When a problem is sliced across many GPUs, the network cables become the fragile arteries of your system. If workers are constantly shouting intermediate results to each other across a slow, narrow hallway, the fastest processors in the world will spend most of their time waiting. Optimizing distributed inference isn't just about speeding up the math; it's about choreographing these massive synchronized dances so that data arrives exactly when the next step requires it.
What to understand
- All-reduce, all-gather, reduce-scatter, all-to-all
- Topology, RDMA, InfiniBand/RoCE, congestion, overlap, and NUMA
Follow the references
1 · Documentation
NVIDIA · NCCL collectives ↗Trace what data moves in each collective and what topology-aware communication means.
2 · Documentation
vLLM · Parallelism and scaling ↗Read multi-node setup and troubleshooting guidance.
Make it concrete
Measure representative collective sizes on the intended topology, then compare collective time with layer compute. Check affinity and link utilization before buying more GPUs.
You understand it when…
Why does low latency matter for many small collectives even on a high-bandwidth network?
This is a study map. Follow the reference for the full explanation; use the question to test your mental model. Browse the reference shelf.