Gradient Descent and Optimization Landscapes

Gradient descent finds a low point of a cost function by repeatedly stepping downhill. It trains every neural network and many robot controllers, and the step size decides whether it creeps, converges, or blows up.

Cost surface Step size Momentum

Click on the map to drop the ball somewhere else.

The update

At each step, move against the gradient $\nabla f$, scaled by the step size $\eta$. Momentum keeps a running velocity $\mathbf{v}$, so steps that agree build up speed and steps that zigzag cancel out:

$$\mathbf{v}_{k+1} = \beta\,\mathbf{v}_k - \eta\,\nabla f(\mathbf{x}_k) \qquad \mathbf{x}_{k+1} = \mathbf{x}_k + \mathbf{v}_{k+1}$$

With $\beta = 0$ this is plain gradient descent.

What to notice

In the round bowl, almost any small step size works. In the narrow valley, the slope across the valley is much steeper than along it, so a step size big enough to make progress along the valley makes the ball bounce from wall to wall. Push the step size up and it diverges. Add momentum and the zigzag calms down while progress along the valley speeds up.

With two minima, where you start decides where you end up. Gradient descent only finds a local minimum, which is why training and trajectory optimization depend so much on a good starting guess.