3.5 Embeddings & Representation
Interview-ready field notes
The 20-second answer
Embeddings are dense vectors learned so items with similar meaning or behavior are near one another in a useful geometric space.
Core ideas to retain
- An embedding layer is a lookup table mapping a discrete ID to a learned continuous vector.
- Similarity search usually compares embeddings with cosine similarity or a dot product; use normalization deliberately.
- Good representations keep task-relevant structure, enabling retrieval, clustering, recommendations, and linear probes.
- Contextual embeddings depend on surrounding text, so the same word can have different vectors in different sentences.
Interview / OA rule
For semantic search: embed documents and query using the same model, normalize if using cosine, retrieve top-, and use metadata/filtering or reranking where needed.
One good written resource
The Illustrated Word2vec — read this after the video when you want a clearer mental model, not more pages of notes.
Most asked
Interview questions to practise aloud
Each answer is the level of detail expected for a strong fundamentals round.
What is an embedding?
A learned dense vector representation of a discrete or complex item. Its geometry captures relationships useful for a task.
Cosine similarity vs Euclidean distance?
Cosine emphasizes direction and ignores magnitude; Euclidean uses both direction and length. Choose the metric the embedding model was trained for.
Static vs contextual embeddings?
Static methods give a word one vector everywhere. Contextual models produce a representation based on the surrounding tokens.