Broad intuition · Study map 23 of 85
Attention: each token asks what matters here
Suggested preparation: Tokens: the model’s units of work
Picture it
Each token makes a query and compares it with other tokens’ keys. The match scores decide how much of each value to mix in. Repeating this across heads and layers lets a token’s representation depend on its context.
What to understand
- Q/K/V, softmax, causal masking, multi-head attention
- MLPs, residual stream, normalization, positions/RoPE, logits
Follow the references
1 · Video
3Blue1Brown · Attention in transformers, step-by-step ↗Follow the query/key matching picture and how it changes a token’s embedding.
2 · Course + videos
Stanford · CS336 Language Modeling from Scratch (2025) ↗Use the architecture lecture for the full transformer block and positional encoding.
Keep the distinction. Attention weights are not, by themselves, a complete explanation of a model’s answer.
You understand it when…
Why must a next-token predictor hide future tokens during training?
This is a study map. Follow the reference for the full explanation; use the question to test your mental model. Browse the reference shelf.