3.3 RNNs / Sequence Models (conceptual)
Interview-ready field notes
The 20-second answer
Sequence models process ordered inputs while carrying context forward. RNNs do this recursively; LSTMs and GRUs add gates so important information lasts longer.
Core ideas to retain
- An RNN updates hidden state , so its output depends on order and prior context.
- Backpropagation through time unfolds the network over tokens. Repeated multiplication can cause vanishing or exploding gradients.
- LSTMs use input, forget, and output gates plus a cell state; GRUs are a simpler gated alternative.
- Transformers usually replaced RNNs for language because attention lets tokens interact in parallel over long context.
Interview / OA rule
If asked why RNNs struggle with long dependencies, say gradients travel through many recurrent steps and decay or explode; gates and attention address this differently.
One good written resource
Colah — Understanding LSTMs — read this after the video when you want a clearer mental model, not more pages of notes.
Most asked
Interview questions to practise aloud
Each answer is the level of detail expected for a strong fundamentals round.
Why does an RNN have vanishing gradients?
Gradients are products across time steps. Repeated factors smaller than one shrink them, so distant tokens barely influence learning.
LSTM vs GRU?
Both are gated RNNs. LSTM has a separate cell state and more gates; GRU combines parts of that design and is simpler.
Why did Transformers displace RNNs in NLP?
Self-attention connects distant tokens directly and processes a sequence in parallel during training, avoiding the strictly sequential recurrence bottleneck.