3.4 Transformers (highest-yield topic right now)
Interview-ready field notes
The 20-second answer
A Transformer lets each token selectively read other tokens with attention, then repeats this with feed-forward layers, residual connections, and normalization.
Core ideas to retain
- Self-attention forms queries, keys, and values. A query scores keys, softmax turns scores into weights, and the weighted values become new context.
- Scaled dot-product attention is . Scaling prevents large dot products from making softmax too sharp.
- Multi-head attention learns several relation types in parallel. Positional encodings tell the model order because attention alone is permutation-invariant.
- Causal masks prevent a next-token model from reading future tokens; encoder-only models such as BERT use bidirectional context.
Interview / OA rule
Know the cost: full self-attention is in sequence length. At generation time, KV caching avoids recomputing old keys and values, but cache memory grows with context.
One good written resource
The Illustrated Transformer — read this after the video when you want a clearer mental model, not more pages of notes.
Most asked
Interview questions to practise aloud
Each answer is the level of detail expected for a strong fundamentals round.
Explain Q, K, and V simply.
A query is what this token seeks, a key is what each token advertises, and a value is the information to retrieve. Query–key similarity chooses how to mix values.
Why divide by $\sqrt{d_k}$?
Dot-product variance grows with dimension. Scaling keeps softmax from saturating, which preserves useful gradients.
Encoder-only vs decoder-only Transformer?
Encoder-only uses bidirectional context for understanding tasks; decoder-only uses a causal mask and predicts the next token for generation.