AI Intuition & LLM Systems

Understand the AI landscape

Recognize the major ideas, explain their mechanism in ordinary language, and know where to go deeper. Read the mental picture, explore the selected part of a reference, then answer the check in your own words.

Run LLMs at scale

Follow a request from the gateway to GPU memory and back. Predict a bottleneck, study the mechanism, run an experiment, and defend the tradeoff. Begin after you can explain tokens, attention, and the generation loop.

First pass: explain the picture without equations. Second pass for serving: predict, measure, and explain a tradeoff. You can start the systems path once tokens, attention, and generation make sense. Start here · Reference shelf · Glossary

AI intuition

01 · The AI landscape

Recognize the major ways machines search, learn, generate, and act. Start with pictures; proofs can wait.

AI, ML, deep learning, and generative AI →

Place classification, search, generation, and control on one map · Distinguish training, inference, and evaluation

Search, logic, and planning →

Breadth-first search, A*, and heuristic quality · Constraint satisfaction, knowledge representation, planning, and minimax

Learning from labels, patterns, and feedback →

Regression versus classification · Supervised, unsupervised, self-supervised, and reinforcement learning

Probability, uncertainty, and causality →

Bayesian inference and probabilistic graphical models · Prediction versus intervention; confounding; randomized experiments

Optimization beyond gradients →

Black-box objectives, random search, evolutionary search · Exploration versus exploitation in hyperparameter tuning

AI intuition

02 · Just enough mathematical intuition

Understand shapes, sensitivity, and uncertainty well enough to read AI explanations without drowning in notation.

Vectors and matrices: coordinates that can move →

Shapes, matrix multiplication, dot products, cosine similarity · Basis changes, projections, SVD, and low-rank approximation

Derivatives: a sensitivity meter →

Partial derivatives and gradients · Chain rule; add contributions when paths meet

Probability and loss: how surprising was the answer? →

Conditional probability, expectation, variance, and Bayes’ rule · Log likelihood, cross-entropy, entropy, perplexity, and KL divergence

AI intuition

03 · How learning succeeds and fails

Build a baseline, recognize overfitting, and choose evaluations that reflect the real task.

Linear models, trees, neighbors, and ensembles →

Linear/logistic regression, k-nearest neighbors, SVMs, and kernels · Decision trees, random forests, gradient boosting, and calibration

Clustering, compression, and anomalies →

k-means, density clustering, PCA, and autoencoders · Distances, scaling, outliers, and the limits of visualization

Generalization: learning the pattern, not the answer sheet →

Train/validation/test splits; leakage; bias and variance · Weight decay, dropout, augmentation, early stopping

Evaluation: the scoreboard shapes the system →

Precision/recall, thresholds, PR and ROC curves, calibration · Temporal/group splits, uncertainty intervals, slice metrics, online experiments

AI intuition

04 · Inside a neural network

Follow a prediction forward, then trace sensitivity backward. This is the foundation for understanding transformers.

Neurons, layers, and representations →

Weights, biases, activations, and nonlinearities · Forward pass, loss, residual connections, and normalization

Backpropagation: trace influence backward →

Loss → output → intermediate activations → every contributing parameter · Reverse-mode automatic differentiation versus the optimizer’s update

Optimizers: walking with noisy directions →

Learning rate, batch size, momentum, Adam, and schedules · Vanishing/exploding gradients, initialization, clipping, and numerical stability

Embeddings: useful coordinates for meaning →

Token, sentence, image, and item embeddings · Contrastive learning, negative examples, transfer learning

AI intuition

05 · Beyond text

Recognize the core ideas in vision, audio, graphs, recommendation, and image generation.

Vision: sliding detectors and image tokens →

Convolution, receptive field, pooling, and translation bias · Classification, detection, segmentation, CNNs, and vision transformers

Sequences: recurrence, attention, and state →

RNNs, LSTMs, temporal dependencies, and forecasting · State-space models, Mamba, and hybrid architectures

Audio, video, and multimodal models →

Spectrograms, speech recognition, speech synthesis · Contrastive image/text alignment, vision-language models, video sampling

Graphs and recommenders: learning from relationships →

Message passing and graph neural networks · Candidate retrieval, ranking, feedback loops, and cold start

VAEs, GANs, diffusion, and flow matching →

Latent variables and reconstruction · Adversarial learning, denoising, sampling steps, and flow trajectories

AI intuition

06 · How LLMs work

Connect tokenization, attention, training, and adaptation to the costs and behavior you will see when serving.

Tokens: the model’s units of work →

BPE, vocabulary, special tokens, Unicode, and chat templates · Input/output tokens, context windows, truncation, and stop tokens

Attention: each token asks what matters here →

Q/K/V, softmax, causal masking, multi-head attention · MLPs, residual stream, normalization, positions/RoPE, logits

Pretraining, instruction tuning, and distillation →

Next-token training, data quality, deduplication, scaling laws · SFT, transfer, catastrophic forgetting, distillation, and data contamination

LoRA: learn a small correction →

Low-rank updates, rank, target layers, merging · QLoRA’s quantized base; training memory versus inference memory

Preferences, rewards, and alignment →

Reward models, reference policies, PPO, DPO, and verifiable rewards · Reward hacking, distribution shift, safety evaluation

Sampling, reasoning, and inference-time compute →

Greedy decoding, temperature, top-k/top-p, beam search · Reasoning budgets, candidate generation, verifiers, and stopping

AI intuition

07 · Acting, retrieving, and knowing the limits

Understand RL, agents, retrieval, causality, and evaluation before building systems around model outputs.

Reinforcement learning: consequences arrive later →

States, actions, rewards, returns, discounting, and MDPs · Exploration, credit assignment, and on/off-policy learning

RL algorithms, imitation, and robotics →

Q-learning, policy gradients, actor-critic, PPO · Model-based planning, imitation, offline RL, sim-to-real

RAG: bring an open book to generation →

Chunking, embeddings, lexical/vector/hybrid search, reranking · Retrieval recall, answer faithfulness, ACLs, freshness, and evaluation

Agents: models inside a control loop →

Tool schemas, structured outputs, state, memory, and orchestration · Timeouts, bounded loops, permissions, retries, and human review

Reliability, fairness, and interpretability →

Hallucination, calibration, distribution shift, and subgroup evaluation · Feature attribution, mechanistic interpretability, privacy, and adversarial inputs

LLMs at scale

08 · One request, end to end

Start the deep path here. Trace work, memory, and latency before choosing an optimization.

The software stack: from tensors to a live process →

Python environments, tensors, shapes, dtype, device, and autograd · Linux processes, host/GPU memory, containers, ports, logs, and driver/runtime compatibility

The life of an inference request →

Gateway → tokenizer → scheduler → model → sampler → stream · Model/tokenizer revisions, chat templates, cancellation, and error handling

Prefill and decode: reading versus writing →

Parallel prompt processing versus sequential generation · Compute, HBM bandwidth, launch overhead, and network bottlenecks

The scoreboard: latency, throughput, and goodput →

TTFT, inter-token latency, time per output token, end-to-end latency · p50/p95/p99, request versus token throughput, SLOs, and goodput

The memory budget: weights are only the beginning →

Weight bytes ≈ parameters × bytes per parameter · KV bytes ≈ 2 × layers × KV heads × head dimension × cached tokens × bytes per element

Establish a baseline you can trust →

Pin model, tokenizer, engine, driver, hardware, and settings · Warmup, repeats, seeds, correctness, and workload distributions

LLMs at scale

09 · Why the GPU waits

Reason from bytes and operations, then use a profiler to test that reasoning.

GPU hardware: workers, shelves, and delivery lanes →

SMs, warps, threads, registers, shared memory, L2, and HBM · SIMT, occupancy, coalescing, tensor cores, precision, and CPU/PCIe input paths

Rooflines: count work per byte moved →

FLOPs/byte and the compute-to-bandwidth ratio · Lower bound: max(FLOPs / FLOPs-per-second, bytes / bytes-per-second)

Profiling: find the empty spaces first →

Nsight Systems timeline; kernel time versus host time · DCGM memory/SM activity, power, clocks, throttling, and error counters

FlashAttention: avoid writing the giant scratchpad →

IO-aware tiling, online softmax, and recomputation · Exact dense attention versus sparse/approximate attention

Fusion and CUDA graphs: less dispatch, fewer round trips →

Kernel fusion, layouts, tiling, and mixed precision · Graph capture, shape buckets, padding, compilation, and warmup

LLMs at scale

10 · Keep useful work on the GPU

Manage growing conversation state and share execution without destroying responsiveness.

KV cache: keep the work you can reuse →

Cache contents per layer; attention reads and cache writes · MHA, MQA, GQA, sliding windows, and cache lifetime

PagedAttention: grow conversations in blocks →

Block tables, internal/external fragmentation, copy-on-write · Allocation, reclamation, sharing, and eviction

Continuous batching: refill seats every iteration →

Iteration-level scheduling, admission, and token budgets · Batch size versus latency, memory, fairness, and throughput

Chunked prefill: stop one reader blocking every writer →

Prefill/decode interference and token budgets · Chunk size, scheduling policy, fairness, and TTFT/ITL tradeoffs

Prefix caching: reuse the shared first pages →

Exact token-prefix reuse, cache keys, model/adapter identity · Hit rate by reused tokens, eviction, privacy, and cache-aware placement

When cache fills: evict, recompute, or move →

Active-request preemption versus reusable-prefix eviction · CPU/remote offload, recomputation, transfer bandwidth, and cache tiers

Long context: fit, attend, and remember usefully →

Full attention versus sliding-window, sparse, and hybrid attention · Context parallelism, RoPE extension, truncation, compaction, and long-context evaluation

LLMs at scale

11 · Spend fewer bytes and steps

Reduce cost while keeping explicit quality and latency gates.

Quantization: fewer bits, different errors →

FP16/BF16, INT8/INT4, FP8, scales, groups, outliers · Weight-only versus weight/activation quantization; calibration and kernel support

KV precision and architectural compression →

KV precision, scale calibration, cache capacity, and attention accuracy · GQA/MQA versus MLA; architecture and backend compatibility

Speculative decoding: draft cheaply, verify together →

Draft cost, acceptance rate, verification cost, and lookahead · Draft models, self-speculation, memory overhead, and load dependence

Smaller models, distillation, and cascades →

Quality/cost Pareto frontier, distillation, task specialization · Confidence calibration, escalation, fallback, and model-version contracts

Structured output: grammar is not truth →

Token masks, grammars, schema complexity, and compilation · Validation, semantic checks, parser versions, and retry budgets

Choose an engine by measured fit →

vLLM, SGLang, TensorRT LLM, and local runtimes · Model/backend compatibility, attention kernels, compile/warmup costs, APIs, and reproducibility

LLMs at scale

12 · Beyond one GPU

Learn which dimension to split and where the resulting communication goes.

Replicate or shard? DP, TP, PP, and CP →

Replicas versus tensor, pipeline, and context parallelism · Per-token collectives, pipeline bubbles, batch size, and failure domains

Interconnects: the slowest road can set the pace →

All-reduce, all-gather, reduce-scatter, all-to-all · Topology, RDMA, InfiniBand/RoCE, congestion, overlap, and NUMA

Mixture of experts: sparse work, uneven traffic →

Total versus active parameters, expert parallelism, all-to-all · Hot experts, load balance, token dispatch, and communication overlap

Separate prefill and decode: two pools, one conversation →

Independent pool sizing, TP layouts, KV transfer, and handoff · Aggregated versus disaggregated serving; goodput under mixed demand

Route to cached work without creating a hotspot →

Least-loaded versus token-work-aware routing · Prefix affinity, stale cache metadata, load estimates, and topology

LLMs at scale

13 · Operate a serving fleet

Handle real traffic, slow starts, failures, and rollout risk without wasting capacity.

Admission control: a bounded waiting room →

Token budgets, deadlines, quotas, backpressure, and load shedding · Fairness, head-of-line blocking, priority, starvation, and output caps

Autoscaling: capacity arrives late →

Queue/work-based signals, hysteresis, warm pools, and scale-down drain · Node provisioning, model download, local caches, and readiness

Deployments: version the entire inference contract →

Immutable artifacts, hashes, provenance, model license/access terms · Startup/readiness/liveness, shadow traffic, canaries, drain, and rollback

Failures: don’t turn one timeout into ten requests →

OOM, GPU errors, worker death, network partitions, and health probes · Exponential backoff with jitter, retry budgets, idempotency, and resumability

Multi-tenancy: share the model, isolate the users →

Adapter loading/eviction, multi-LoRA batching, and cache identity · Per-tenant quotas, accounting, noisy neighbors, MIG and time sharing

LLMs at scale

14 · Serve whole AI applications

Generation is one stage. Retrieval, tools, multimodal encoders, and sessions create their own scaling problems.

Retrieval at scale: search has its own budget →

HNSW/IVF/PQ intuition, recall-latency tradeoffs, hybrid retrieval · Chunk/version IDs, incremental indexing, deletes, ACL filtering, and reranking

Agent workloads: one user can create many requests →

Session affinity, context growth, tool latency, and parallel branches · Task budgets, durable state, cancellation, idempotent tools, and loop detection

Multimodal serving: text tokens miss part of the bill →

Resolution/frame sampling, encoder batching, and embedding transfer · Separate encoder/prefill/decode pools, cache keys, and input limits

Streaming: deliver tokens without losing control →

SSE/WebSocket behavior, proxy buffering, and time-to-visible-token · Connection limits, cancellation, queue bounds, and stream errors

Offline batches, local models, and heterogeneous fleets →

Offline queues, checkpointed work units, spot/preemptible capacity · CPU/unified memory, GPU/TPU portability, local inference, and privacy

LLMs at scale

15 · Training systems you need to recognize

Know how models and adapters are produced, and why training has a different resource profile from inference.

Training memory: keep enough history to learn →

Parameters, gradients, optimizer states, and saved activations · Micro-batches, gradient accumulation, mixed precision, and activation checkpointing

Distributed training: synchronize learning →

DDP, all-reduce, FSDP/ZeRO stages, tensor and pipeline parallelism · Global batch, accumulation, synchronization, and communication overlap

Data pipelines and checkpoints: keep GPUs fed and work recoverable →

Shuffling, packing, tokenization, data provenance, and input stalls · Distributed checkpoints, consistency, storage bandwidth, and resume validation

RL post-training: alternate generating and learning →

Rollout engines, reward/verifier workers, learner, and weight transfer · Synchronous versus asynchronous updates, policy lag, and capacity sharing

LLMs at scale

16 · Measure, protect, and pay for the service

Connect performance work to correctness, observability, security, and business cost.

Observability: connect the user symptom to the machine →

Request traces, phase histograms, SLO/error budgets, and burn rates · Queue length, active tokens, preemption, cache hits, GPU/network health

Load testing: recreate the traffic, not just the average →

Open versus closed loop, trace replay, warm/cold caches · Arrival process, length distributions, steady state, errors, and tail metrics

Quality gates: fast wrong answers are wasted work →

Golden sets, task slices, retrieval/agent evaluation, and regressions · LLM-judge bias, human calibration, contamination, and release gates

Security: treat inputs and artifacts as untrusted →

Prompt injection, tool permissions, tenant/cache isolation, and data retention · Model supply chain, remote code, parser/media limits, secrets, and audit trails

Capacity and cost: price successful work →

Cost per successful task and per million input/output tokens · Little’s law, service demand, headroom, reservations, and failure capacity

LLMs at scale

17 · Build evidence of engineering judgment

Small reproducible investigations, then one end-to-end serving design. GPU experiments can use a small model; distributed extensions need suitable hardware.