Human Pose and Hand Tracking

Pose estimators find a person's keypoints, such as shoulders, elbows, and wrists, in each image, and draw a skeleton between them. Robots use this to follow people, understand gestures, and learn from human demonstrations. Keypoints get noisy or vanish when a limb is hidden or moves fast.

Motion Occlusion of the right arm Motion speed Smoothing

Keypoints and confidence

Models like OpenPose, MediaPipe, and ViTPose output, for each keypoint $k$, an image position $\mathbf{u}_k$ and a confidence $c_k$. Low-confidence points are dropped. A simple smoother blends each new detection with the previous estimate:

$$\hat{\mathbf{u}}_t = \alpha\,\hat{\mathbf{u}}_{t-1} + (1 - \alpha)\,\mathbf{u}_t$$

What to notice

When the arm is hidden, its keypoints drop out or jump to wrong spots, so the skeleton breaks. Fast motion adds blur, which adds noise. Smoothing calms the jitter, but too much makes the skeleton lag behind the real motion, which matters if a robot is reacting to a reaching hand. Hands are harder still: 21 keypoints each, small in the image, and often hidden by the hand itself.