Engineering reading desk
Follow maintainers and researchers for mechanisms you can test. Start with the linked lessons so each article has a purpose.
Model and hardware reasoning
Read the scaling book with a scratchpad: calculate bytes, FLOPs, and synchronization before accepting a speed claim.
Book chapter · How To Scale Your Model · Rooflines ↗
Book chapter · How To Scale Your Model · Transformer inference ↗
Book chapter · How To Scale Your Model · Sharding ↗
Serving engine behavior
Use maintainers’ documentation and examples to understand the scheduler and cache policies in the version you run.
- Continuous batching: refill seats every iteration →
- Prefix caching: reuse the shared first pages →
- Structured output: grammar is not truth →
Documentation · vLLM · Optimization and tuning ↗
Documentation · SGLang · Serving documentation ↗
Fleet behavior
Connect routing, warm capacity, and overload to what a user sees.
- Route to cached work without creating a hotspot →
- Autoscaling: capacity arrives late →
- Admission control: a bounded waiting room →
Documentation · NVIDIA Dynamo · Router design ↗
Documentation · Ray Serve · Advanced autoscaling ↗
Book chapter · Google SRE · Handling overload ↗
Model-controlled workflows
Look for a clear reason to add another model call and a clear boundary for what it can do.
Engineering article · Anthropic · Building effective agents ↗