LLM system design: from workload to fleet
A design starts with request sizes, arrival patterns, quality needs, latency targets, and failure expectations. Choose mechanisms only after locating a constraint.
1 · Define the contract
Separate prompt work from generation work. Specify time to first token, stream latency, and successful tasks per unit cost.
2 · Fit and schedule the work
Budget cache, admit bounded work, and prevent long prompts from stopping active streams.
3 · Scale the measured bottleneck
Replicate, shard, or separate phases based on the workload and communication costs.
4 · Keep the service dependable
Plan warm capacity, overload behavior, isolation, rollouts, and recovery.
5 · Defend the result
Deliver a reproducible design with a measured baseline, explicit assumptions, and the next experiment that might change your choice.