Cross-Embodiment Learning

Every robot lab collects data on its own robot, and no single one has enough. Cross-embodiment learning trains one policy on data from many robots: arms of different makes, mobile manipulators, even different grippers. Their actions are mapped into a shared form, such as end-effector motions, so the policy can learn what the tasks have in common.

Include data from a similar arm Include data from a different arm brand Include data from a mobile manipulator Include data from a robot with a suction gripper Demonstrations on the new robot

Shared action spaces

A policy trained across robots needs an action format they all share. A common choice is a change in the end-effector pose plus a gripper command, $\mathbf{a} = (\Delta\mathbf{p},\ \Delta\mathbf{R},\ g)$, with each robot's own controller turning that into joint motions. Data from other robots then acts like extra data for the new one, weighted by how similar they are. A simple way to picture the effect:

$$n_{\text{effective}} = n_{\text{own}} + \sum_{k} s_k\, n_k \qquad 0 \le s_k \le 1 \text{ (similarity)}$$

What to notice

With pooled data, the new robot reaches a high success rate with far fewer of its own demonstrations. Similar robots help most. Very different ones, such as a suction gripper on a grasping task, help little and can even hurt by adding conflicting examples, shown here as a small negative weight. The curves on this page are an illustrative model of that effect, not measured results. Datasets such as Open X-Embodiment, with data from over twenty robot types, and the models trained on them showed this transfer in practice.