Extrinsic Calibration: Camera to LiDAR and Hand to Eye

To fuse two sensors, the robot needs to know exactly where one sits relative to the other. That transform is called the extrinsic calibration. A small error in it means camera pixels land on the wrong LiDAR points.

Guess: camera x Guess: camera y Guess: camera rotation Measurement noise

The problem

Both sensors see the same target corners. The LiDAR reports them as $\mathbf{p}_i$ in its frame and the camera as $\mathbf{q}_i$ in its frame. The extrinsic calibration is the rotation $R$ and translation $\mathbf{t}$ that best carry one set onto the other:

$$\min_{R,\,\mathbf{t}}\ \sum_i \left\lVert \mathbf{p}_i - (R\,\mathbf{q}_i + \mathbf{t}) \right\rVert^2$$

With known matches, this has a closed-form answer: center both point sets, find the rotation from the SVD of their cross-covariance, then the translation. This is the Kabsch method.

Hand to eye

A camera mounted on a robot arm needs the same thing relative to the gripper. There the robot moves to several poses $A_i$, the camera sees a fixed target and measures $B_i$, and the unknown camera mount $X$ satisfies $A X = X B$ for every pair of motions.

What to notice

Try to line up the points with the sliders. It is fiddly, and a small rotation error grows with distance from the sensor. Press "Solve" for the exact answer. With noise, a single target pose gives a shaky result. Adding more target poses, spread around the field of view, makes the answer more reliable.