Tools by the job they do
Learn one serving engine well enough to explain its behavior. Use alternatives to test a specific architectural or hardware tradeoff.
Serve and schedule
vLLM and SGLang provide engines for batching, cache management, and generation. Start with one reproducible small-model baseline.
- Establish a baseline you can trust →
- Continuous batching: refill seats every iteration →
- Choose an engine by measured fit →
Documentation · vLLM · Optimization and tuning ↗
Documentation · SGLang · Serving documentation ↗
Documentation · NVIDIA · TensorRT LLM overview ↗
Coordinate a fleet
Dynamo adds routing and distributed serving mechanisms. Ray Serve provides deployment and autoscaling machinery. Understand the engine/fleet boundary.
Documentation · NVIDIA · Dynamo ↗
Documentation · Ray Serve · Advanced autoscaling ↗
See and change GPU work
Use Nsight for timelines, DCGM for fleet/hardware signals, and Triton for learning kernel dataflow.
Documentation · NVIDIA · Nsight Systems user guide ↗
Documentation · NVIDIA · DCGM feature overview ↗
Tutorials · Triton · GPU programming tutorials ↗
Evaluate the full task
Use an evaluation harness for repeatable model tests and a load generator for service behavior. Add task-specific checks where a benchmark does not fit.
- Quality gates: fast wrong answers are wasted work →
- Load testing: recreate the traffic, not just the average →
Code + guide · EleutherAI · Language Model Evaluation Harness ↗
Documentation · vLLM · Serving benchmark reference ↗
Retrieve or run locally
Faiss studies the retrieval-index tradeoff; llama.cpp makes local inference and hardware constraints tangible.
- Retrieval at scale: search has its own budget →
- Offline batches, local models, and heterogeneous fleets →
Documentation · Meta · Faiss similarity search ↗
Code + guide · ggml · llama.cpp ↗