Learning from Human Videos

There are billions of hours of video of people handling things, far more than any robot could ever collect. Robots cannot copy human motions directly, since their bodies differ, but videos still show where and how objects are used: mugs are held by the handle, drawers pulled by the knob. Learning these affordances is one way to use human video for robots.

Videos watched Hand tracking noise Object

Affordances from video

A hand tracker finds where a person's fingers touch an object in each video, with some error. Adding up many contacts gives a heat map $h(\mathbf{x})$ of where the object is usually held. The robot grasps where the map is strongest, among spots its gripper can reach:

$$\mathbf{x}_{\text{grasp}} = \arg\max_{\mathbf{x}} \sum_{k=1}^{N} \exp\!\Big(-\frac{\lVert\mathbf{x} - \mathbf{c}_k\rVert^2}{2\sigma^2}\Big)$$

What to notice

With a few videos, the heat map is patchy and the robot might grab the mug's body or its rim. With more, the handle clearly stands out. Noise in hand tracking blurs the map but does not move its peak much, because errors average out. Real systems learn this with neural networks trained on large egocentric video datasets, and also learn the motions that follow, such as pulling a drawer outward rather than up, then retarget them to the robot.