Deep systems understanding · Study map 63 of 85
Autoscaling: capacity arrives late
Suggested preparation: Admission control: a bounded waiting room
Picture it
Summoning a new server isn't like flipping a light switch; it's like waking a giant from a deep slumber. The GPU has to slowly stretch, download massive weight files, and warm up its internal pathways before it can answer a single question. If you wait to signal for help until the room is already overflowing with tasks, the new capacity will arrive too late to prevent a crisis, forcing you to predict traffic waves before they crest.
What to understand
- Queue/work-based signals, hysteresis, warm pools, and scale-down drain
- Node provisioning, model download, local caches, and readiness
Follow the references
1 · Documentation
Ray Serve · Advanced autoscaling ↗Use scale-up/down delays and in-flight request guidance to design a stable control loop.
2 · Documentation
Kubernetes · Schedule GPUs ↗Connect GPU scheduling to available node resources and placement.
Make it concrete
Measure cold-start stages separately. Replay a burst; compare reactive scaling with a warm minimum. Record wasted idle cost and SLO violations, not just replica count.
You understand it when…
Why can aggressively adding and removing replicas lower cache hit rate and increase latency?
This is a study map. Follow the reference for the full explanation; use the question to test your mental model. Browse the reference shelf.