Bundle Adjustment

After a robot tracks features through many images, it has rough estimates of where each camera was and where each point is. Bundle adjustment adjusts all of them at once so that every point lines up with every image that saw it. The name comes from the bundles of light rays between cameras and points.

Starting error

The cost

Each camera $j$ with pose $T_j$ measures the bearing $z_{ij}$ to point $\mathbf{p}_i$. Bundle adjustment minimizes the total squared difference between what each camera measured and what the current estimate predicts:

$$\min_{\{T_j\},\,\{\mathbf{p}_i\}}\ \sum_{i,j} \big\lVert z_{ij} - \pi(T_j, \mathbf{p}_i) \big\rVert^2$$

It is solved with Gauss-Newton or Levenberg-Marquardt. This page works in 2D, with bearings instead of pixels, to keep it visible.

What to notice

The error drops sharply in the first few iterations as cameras and points pull each other into place. The first camera is held fixed, because otherwise the whole scene could slide or turn without changing any measurement, a freedom called the gauge. With bearings alone the overall scale is also unknown, so the baseline between the first two cameras is held at its true length. Real systems, from photogrammetry to visual SLAM, run bundle adjustment on thousands of cameras and millions of points, using the sparse structure of the problem to stay fast.