Capacity and cost: price successful work

Picture it

Running a restaurant isn't just about the cost of the stove. It's the rent for the dining room, the ingredients in the fridge, the waitstaff during slow hours, and the meals comped for mistakes. True capacity planning looks at the entire price of delivering a successful dish during the busiest hour, leaving a margin for dropped plates and sudden rushes.

What to understand

• Cost per successful task and per million input/output tokens
• Little’s law, service demand, headroom, reservations, and failure capacity

Follow the references

1. 1 · Book chapter

   How To Scale Your Model · Transformer inference ↗

   Use latency/throughput tradeoffs as the basis of a capacity model.

2. 2 · Book chapter

   Google SRE · Handling overload ↗

   Connect utilization targets to overload behavior and reserve capacity.

Make it concrete

Use L ≈ λW for a stable system as a concurrency sanity check, then size from measured goodput per replica. Add a replica-failure scenario and calculate cost with explicit utilization assumptions.