Broad intuition · Study map 22 of 85
Tokens: the model’s units of work
Suggested preparation: VAEs, GANs, diffusion, and flow matching
Picture it
Text is cut into reusable pieces and converted to integer IDs. The model sees these pieces, not words as humans define them. A tokenizer choice changes sequence length, costs, and which patterns are easy to learn.
What to understand
- BPE, vocabulary, special tokens, Unicode, and chat templates
- Input/output tokens, context windows, truncation, and stop tokens
Follow the references
1 · Video
Stanford CS336 · Overview and tokenization ↗Watch the tokenization portion and follow one merge example.
2 · Course
Hugging Face · LLM course ↗Use the tokenizer chapter to compare text with its token IDs.
You understand it when…
Why can equally long English text, code, and another language produce different token counts?
This is a study map. Follow the reference for the full explanation; use the question to test your mental model. Browse the reference shelf.