Replicate or shard? DP, TP, PP, and CP

Picture it

Imagine a factory building a complex machine. You could clone the entire assembly line to build multiple machines at once, or you could have workers simultaneously tighten different bolts on the same part. Alternatively, you could form a bucket brigade where each worker passes the half-finished product to the next room. Each choice stretches the work across different physical spaces, shifting the bottleneck from raw computation to the conveyor belts connecting the workers.

What to understand

• Replicas versus tensor, pipeline, and context parallelism
• Per-token collectives, pipeline bubbles, batch size, and failure domains

Follow the references

1. 1 · Documentation

   vLLM · Parallelism and scaling ↗

   Use the deployment guidance to connect model size and topology to parallelism choices.

2. 2 · Book chapter

   How To Scale Your Model · Sharding ↗

   Draw what each device holds and what it exchanges at every layer.

Make it concrete

Sketch weight/cache placement for DP=2, TP=2, and PP=2. Predict communication and latency; benchmark the feasible configurations on identical traffic.