A small picture for each term
135 intuitive definitions. Follow a term into its lesson for the mechanism, references, and a question to test your understanding.
135 terms
- Activation →
- A signal computed for a particular input; unlike a learned weight, it changes from example to example.
- Actor-critic →
- An RL setup with a policy that acts and a value estimator that helps guide its updates.
- Admission control →
- Deciding which work to accept before it overwhelms finite queues, memory, or service capacity.
- Agent →
- A model-driven loop that can choose actions or tools and use their results to decide what to do next.
- AI →
- The broad field of building systems that search, reason, learn, perceive, or act toward goals.
- All-reduce →
- Combining values from devices, such as summing gradients, and delivering the combined result to all participants.
- Attention →
- Mixing value vectors according to matches between a query and keys, making representations context-dependent.
- Backpressure →
- Signaling or enforcing that upstream producers slow down when downstream capacity is exhausted.
- Backpropagation →
- Efficiently applying the chain rule backward to calculate loss gradients, including for parameters far from the output.
- Bayesian optimization →
- Using a model of an expensive objective to choose informative or promising next experiments.
- Cache-aware routing →
- Choosing a worker using reusable cached state as well as load and other placement costs.
- Calibration →
- Agreement between stated probabilities and observed frequencies over comparable predictions.
- Canary →
- A limited release used to compare a candidate against a baseline before wider deployment.
- Causal inference →
- Reasoning about what changes under an intervention, beyond correlations in observed data.
- Causal mask →
- A restriction that prevents a token from attending to future tokens in autoregressive training or generation.
- Chain rule →
- Combining local sensitivities along a computation path to find the sensitivity of the final result.
- Chat template →
- The exact formatting and special tokens used to turn conversation roles and messages into model input.
- Checkpoint →
- Saved state for recovery; resuming training may require optimizer, scheduler, RNG, and data-position state in addition to weights.
- Chunked prefill →
- Splitting prompt processing into smaller scheduled pieces so it can share time with ongoing generation.
- Clustering →
- Grouping examples according to a chosen notion of similarity, which depends on representation and scale.
- Cold start →
- The time spent provisioning, loading, initializing, and warming a worker before it is ready to serve.
- Collective →
- A coordinated multi-device communication operation such as all-reduce, all-gather, or all-to-all.
- Compute-bound →
- A regime where arithmetic work limits speed more than data movement or other overheads.
- Constrained decoding →
- Restricting possible next tokens to enforce a grammar or schema, without guaranteeing factual correctness.
- Context parallelism →
- Partitioning sequence-related computation/state across devices, trading local memory or work for communication.
- Continuous batching →
- Admitting and removing requests between generation iterations instead of waiting for an entire fixed batch to finish.
- Convolution →
- Applying shared local filters across an input, making the same detector reusable at different positions.
- Cross-entropy →
- A loss that penalizes assigning low probability to the observed target.
- CUDA graph →
- A captured GPU work sequence that can be replayed with less host dispatch overhead.
- Data leakage →
- Letting information into training or validation that would not be available for the real prediction task.
- Data parallelism →
- Serving independent requests on replicated model workers; in training, replicas also synchronize learning updates.
- Decision tree →
- A prediction procedure built from a sequence of feature-based questions.
- Decode →
- Repeatedly processing generated tokens and cached history to produce subsequent tokens.
- Diffusion model →
- A generative model that learns to reverse a noising process, often generating through repeated denoising steps.
- Distillation →
- Teaching a student model from a teacher’s outputs or intermediate behavior, often to reduce inference cost.
- Distribution shift →
- A change between the data or situations used to develop a model and those it encounters later.
- Dot product →
- A sum of paired products that measures alignment, influenced by both direction and length.
- DPO →
- Learning from preferred and dispreferred responses through a direct preference objective.
- Embedding →
- A learned vector representation whose useful relationships depend on the training objective.
- Ensemble →
- Several predictors combined so their collective behavior can outperform an individual model.
- Entropy →
- The average uncertainty or surprise in a probability distribution.
- Evolutionary search →
- Improving a population of candidate solutions by selecting and varying promising ones.
- Expert parallelism →
- Placing different experts on different devices and dispatching tokens to the devices that hold their selected experts.
- FlashAttention →
- Tiled attention that reduces HBM traffic and avoids materializing the full score matrix, without changing dense attention’s quadratic arithmetic.
- Flow matching →
- Training a model to predict a velocity field that transports samples between distributions.
- FSDP / ZeRO →
- Sharding training state across devices to reduce replication, with communication and resharding costs.
- GAN →
- A generator learns to produce examples using feedback from a discriminator trying to distinguish them from real data.
- Goodput →
- Work completed within defined service targets; the target and counting unit must be explicit.
- GQA →
- Grouped-query attention: multiple query heads share a smaller set of key/value heads, reducing KV storage and traffic.
- Gradient →
- The list of local sensitivities of a scalar output, such as loss, to its input variables or parameters.
- Gradient accumulation →
- Combining gradients from multiple micro-batches before an optimizer step to form a larger effective batch.
- Gradient boosting →
- Adding models that improve the current ensemble according to the loss gradient.
- Graph neural network →
- A model that updates representations using graph structure and information from neighboring nodes.
- Hallucination →
- An output that asserts unsupported or false information despite appearing plausible.
- HBM →
- High-bandwidth memory attached to an accelerator; capacity limits what fits, bandwidth limits how fast data can be moved.
- Heuristic →
- A cheap estimate that helps search choose promising paths; its quality affects both speed and guarantees.
- Idempotency →
- Repeated execution of the same operation having the intended effect only once, important for safe retries of actions.
- ITL →
- Inter-token latency: elapsed time between successive streamed tokens; its distribution reveals pauses hidden by averages.
- Kernel fusion →
- Combining operations to reduce launches and avoid storing intermediate results in slower memory.
- KL divergence →
- A directional measure of how one probability distribution differs from another.
- KV cache →
- Stored keys and values for previously processed tokens, avoiding recomputation while retaining the cost of reading relevant history.
- KV offload →
- Moving cache state out of GPU memory and paying transfer cost when it is needed again.
- Learning rate →
- The scale of a parameter update; too large can destabilize learning and too small can slow it.
- Little’s law →
- In a stable system, average work in flight equals average arrival rate times average time in the system: L = λW.
- Load shedding →
- Rejecting or dropping selected work so overload does not make the whole service collapse.
- Logits →
- Unnormalized scores over possible next tokens, converted into probabilities for selection.
- LoRA →
- Learning a low-rank weight correction while keeping the base weights frozen.
- Low rank →
- A transformation that can be expressed through fewer intermediate dimensions, allowing a compact factorization.
- Machine learning →
- Fitting behavior from examples or feedback instead of specifying every rule by hand.
- Matrix →
- A rectangular array that can represent a linear transformation from one coordinate system to another.
- MIG →
- NVIDIA Multi-Instance GPU: hardware partitioning of supported GPUs into isolated resource instances.
- MLA →
- Multi-head latent attention: an architecture using compressed latent representations to reduce attention-cache cost.
- MoE →
- Mixture of experts: routing each token through selected experts, reducing active computation but retaining substantial total state.
- MQA →
- Multi-query attention: query heads share one key head and one value head per layer.
- Multimodal model →
- A model that processes or combines more than one kind of input or output, such as text, images, and audio.
- Open-loop load test →
- Offering requests independently of completion so slowdown does not automatically reduce the offered rate.
- Optimizer →
- The rule that uses gradients and possibly stored statistics to update parameters; separate from gradient calculation.
- Overfitting →
- Learning details of the training examples that do not transfer well to new examples.
- PagedAttention →
- An attention/cache design that maps logical token positions to separately allocated physical KV blocks.
- Parameter →
- A learned setting of a model, such as a weight or bias.
- PCA →
- Rotating centered data toward directions of greatest variance, often keeping only the leading directions.
- Perplexity →
- Exponentiated average negative log likelihood; interpret comparisons only with compatible tokenization and evaluation data.
- Pipeline parallelism →
- Splitting model layers into stages on different devices; scheduling and bubbles affect utilization.
- Policy →
- The rule or probability distribution an agent uses to choose actions.
- Precision and recall →
- Precision asks how many predicted positives are correct; recall asks how many actual positives were found.
- Preemption →
- Interrupting an active request to make room; later continuation may require recomputing or restoring its state.
- Prefill →
- Processing prompt tokens and building their cache, producing the logits used to select the first generated token.
- Prefix caching →
- Reusing previously computed state for an exact compatible token prefix, saving repeated prefill work.
- Pretraining →
- The broad initial learning stage, typically over large datasets and a general prediction objective.
- Prompt injection →
- Untrusted content attempting to redirect model behavior; software permissions must bound any resulting actions.
- QLoRA →
- Training low-rank adapters through a frozen quantized base model; it still requires that base and training activations.
- Quantization →
- Using a reduced set of representable numerical values to save memory or accelerate supported computation, with error and kernel tradeoffs.
- RAG →
- Retrieving external evidence and supplying it to a generator; evidence quality and faithful use must be tested separately.
- RDMA →
- Remote direct memory access: moving data between machines with reduced CPU involvement, subject to fabric and software support.
- Recommender →
- A system that retrieves and ranks items for a user or context, often in several stages.
- Regularization →
- Biasing learning toward some solutions, often to improve generalization or control complexity.
- Reranking →
- Applying a more expensive scoring model to a smaller retrieved candidate set.
- Residual connection →
- A path that adds an earlier representation to a transformed version, helping information and gradients pass through depth.
- Retry amplification →
- Extra requests caused by retries, potentially turning a partial failure into overload.
- Reward hacking →
- Improving a measured reward by exploiting its shortcomings instead of achieving the intended objective.
- RLHF →
- Using human feedback, often through a learned reward model, to improve a policy’s outputs.
- Rollout →
- A sequence of actions or generated tokens produced by a policy, used as experience for learning or evaluation.
- Roofline model →
- A performance bound combining peak computation with memory bandwidth and arithmetic intensity.
- RoPE →
- Rotary position embeddings rotate query/key coordinates so attention can depend on relative position.
- SFT →
- Supervised fine-tuning: learning from demonstrations of desired input-output behavior.
- SLO →
- A service-level objective such as a latency or success-rate target for a defined workload and time window.
- SM →
- A GPU streaming multiprocessor that schedules and executes groups of threads using local compute and storage resources.
- Speculative decoding →
- Drafting tokens cheaply and verifying several together with a target model; gains depend on acceptance and overhead.
- State-space model →
- A sequence model organized around an evolving state; selective variants choose how information enters and persists.
- Tail latency →
- The slow end of a latency distribution, commonly described by p95 or p99 rather than only an average.
- Temperature →
- A scaling of logits that changes how concentrated the sampling distribution is.
- Tensor core →
- Specialized hardware for supported matrix operations and number formats.
- Tensor parallelism →
- Splitting a layer’s tensor computation across devices, typically requiring communication within layers.
- Throughput →
- Work completed per time, with the unit stated explicitly: requests, input tokens, output tokens, or tasks.
- Token →
- A model’s discrete unit of text, which may be a word piece, character, byte-based piece, or special marker.
- Top-p →
- Sampling from a set of likely tokens whose cumulative probability reaches a chosen threshold.
- TPOT →
- Time per output token, often an average per request after the first token; verify the benchmark’s exact definition.
- TTFT →
- Time to first token: define whether measured from client submission or server receipt, and include the relevant queueing and processing.
- Unsupervised learning →
- Looking for structure without a supplied answer key; the structure need not match human categories.
- VAE →
- A generative model that learns a probabilistic latent representation and a decoder back to data.
- Value function →
- An estimate of future return from a state or state-action pair under specified assumptions.
- Vector →
- An ordered set of coordinates; embeddings use vectors to represent items in a learned space.
- Warp →
- A group of GPU threads scheduled together under the hardware’s execution model.