Time Sync, Latency, and Sensor Timestamps
Every sensor reading describes the world at one instant. If a camera image takes 80 ms to arrive but is stamped with its arrival time, the robot will think it saw things from a pose it was no longer in. On a turning robot, that small delay becomes a large position error.
The error from a delay
If the robot turns at rate \(\omega\) and a reading is late by \(\Delta t\) but stamped as current, its direction is off by \(\omega\,\Delta t\). At distance \(d\), a landmark appears shifted sideways by about
What to notice
At 45 °/s with an 80 ms delay the angle error is only 3.6°, but a landmark 8 m away lands half a meter from where the LiDAR sees it. The error grows with turn rate, delay, and distance. Check "Stamp images with their true capture time" and the two sensors agree again, because the fusion now looks up the robot's pose at the moment the image was really taken.
Real robots handle this with hardware timestamps from the sensor, clock sync between computers using NTP or PTP, and buffers of past poses, which in ROS 2 is what TF's time-aware lookup does. Sensors on separate computers can also disagree on what time it is, which causes the same kind of error.