3.1 Neural Network Basics

The 20-second answer

A neural network stacks learned linear transforms with nonlinear activations, then learns the weights with backpropagation and gradient descent.

Core ideas to retain

• One layer computes $z=Wx+b$, then an activation such as ReLU or GELU adds nonlinearity.
• Without activations, stacked linear layers collapse into one linear transform and cannot learn nonlinear boundaries like XOR.
• The forward pass predicts; the loss scores that prediction; the backward pass computes gradients; the optimizer updates weights.
• Initialization and normalization keep activations and gradients in a workable range as depth grows.

Interview / OA rule

For a basic OA question, track tensors by shape: batch $B$, input features $d_{in}$, output features $d_{out}$; $X[B,d_{in}]W[d_{in},d_{out}]$ gives $[B,d_{out}]$.

One good written resource

Google — Neural Networks — read this after the video when you want a clearer mental model, not more pages of notes.