4.4 Inference-time Concepts
Interview-ready field notes
The 20-second answer
Inference turns a prompt into tokens and generates one token at a time; decoding settings control the quality–diversity–latency trade-off.
Core ideas to retain
- Autoregressive decoding repeatedly predicts the next-token distribution, chooses a token, appends it, and repeats.
- Temperature rescales logits: lower is more deterministic; higher spreads probability and adds variety.
- Top- samples only among the highest-probability tokens; top- samples the smallest set whose cumulative probability reaches .
- KV cache stores prior attention keys and values, avoiding repeated work for the existing prefix during generation.
Interview / OA rule
For factual or structured output, start with low temperature. For creative alternatives, raise temperature modestly and use top-. Sampling changes outputs, not what the model learned.
One good written resource
Hugging Face — Generation Strategies — read this after the video when you want a clearer mental model, not more pages of notes.
Most asked
Interview questions to practise aloud
Each answer is the level of detail expected for a strong fundamentals round.
Temperature vs top-p?
Temperature reshapes the whole probability distribution; top-p truncates it to a variable high-probability set. They can be used together.
What is a KV cache?
A cache of keys and values for already generated tokens. It speeds decoding but consumes memory proportional to layers, context length, and batch size.
Why is LLM generation slow?
Each new token depends on prior tokens, so decoding is sequential. Batching, KV caching, optimized kernels, and speculative decoding reduce the cost.