Deep systems understanding · Study map 69 of 85
Multimodal serving: text tokens miss part of the bill
Suggested preparation: Agent workloads: one user can create many requests
Picture it
When a model processes a high-definition video, it's not just reading a few words; it's swallowing a massive, dense block of visual data all at once. The text generation might be a lightweight trickle, but the initial visual encoding hits the system like a tidal wave, stressing entirely different bottlenecks like CPU preprocessing and network bandwidth. Balancing this requires recognizing that visual tokens have a completely different weight and rhythm than language tokens, reshaping the entire flow of the pipeline.
What to understand
- Resolution/frame sampling, encoder batching, and embedding transfer
- Separate encoder/prefill/decode pools, cache keys, and input limits
Follow the references
1 · Documentation
NVIDIA · Dynamo ↗Find encode-prefill-decode disaggregation and trace its extra handoff.
2 · Course
Hugging Face · Audio course ↗Connect audio representation and chunking to streaming latency.
Make it concrete
Split the timeline into input decoding, encoding, prefill, and generation. Sweep image resolution or audio duration; measure each stage and output quality.
You understand it when…
Why might adding language-model GPUs fail to improve an image-heavy service?
This is a study map. Follow the reference for the full explanation; use the question to test your mental model. Browse the reference shelf.