Neural Networks 101

A neural network is a stack of simple units. Each one adds up its inputs with weights, then passes the result through a bend, such as tanh. Training nudges all the weights to reduce the error on examples. Even a tiny network can learn curved boundaries that no straight line could draw.

Data Hidden units Learning rate

The network

This network has two inputs, one hidden layer of $n$ tanh units, and one output squashed to a probability with the sigmoid $\sigma$:

$$\mathbf{h} = \tanh(W_1\mathbf{x} + \mathbf{b}_1) \qquad \hat y = \sigma(\mathbf{w}_2^\top \mathbf{h} + b_2) \qquad L = -\big[y\log\hat y + (1-y)\log(1-\hat y)\big]$$

Training uses gradient descent: backpropagation computes how the loss changes with every weight, and each weight takes a small step downhill.

What to notice

With one hidden unit the network can only draw a nearly straight boundary, so it fails on the circle and XOR. Add units and the boundary bends to fit. A learning rate that is too high makes the loss jump around, and one too low learns very slowly. This same recipe, scaled up to billions of weights, is what trains vision models and robot policies.