Gesture and Intent Recognition

Robots working with people need to understand them without words: a wave, a point, a "stop" gesture, or simply which object a hand is moving toward. Recognizing intent early lets a robot hand over a tool, get out of the way, or prepare to help, before the person finishes moving.

Mode Reaching noise

In gesture mode, draw a circle, a check mark, a zigzag, or a straight line.

Template matching

A simple and surprisingly good gesture recognizer, in the style of the $1 recognizer, resamples the drawn stroke to a fixed number of points, scales and centers it, then compares it with each template by the average distance between matching points:

$$d(A, B) = \frac{1}{N}\sum_{i=1}^{N} \lVert \mathbf{a}_i - \mathbf{b}_i \rVert \qquad \text{label} = \arg\min_{\text{templates}} d$$

Early intent prediction

For reaching, the robot updates a probability for each target as the hand moves, favoring targets that the hand's motion points toward, like the goal inference used in shared autonomy.

What to notice

Gesture matching works for clear strokes but confuses similar shapes when drawn sloppily, and it knows only the templates it has. For reaching, the right target usually becomes likely well before the hand arrives, which is the useful part, but targets in nearly the same direction stay ambiguous until late. Learned models from body tracking do both tasks today, with the same trade-off between acting early and being sure.