Imitation Learning: Behavior Cloning

Behavior cloning is the most direct way to teach a robot: record an expert doing the task, then train a policy to map what the robot sees to what the expert did. It is just supervised learning. The catch is that the robot's own small mistakes take it to places the expert never went, where the policy has no good data.

Number of demonstrations Noise when the policy runs

Cloning the expert

Collect pairs of states and expert actions $(s_i, a_i^*)$ from demonstrations, then fit a policy by supervised learning. Here the policy is a simple nearest-neighbor regressor:

$$\pi^* = \arg\min_{\pi}\ \sum_i \big\lVert \pi(s_i) - a_i^* \big\rVert^2$$

Compounding errors

At run time, a small error moves the robot slightly off the demonstrated states, where the policy is less accurate, which causes a bigger error. Over a task of $T$ steps, errors can grow roughly as $T^2$ times the per-step error.

What to notice

With one demonstration and some noise, the policy follows the path at first, then drifts off and hits the obstacle or wanders. More demonstrations cover more of the states the robot might reach, so it recovers more often. Adding noise during demonstrations, so the expert shows how to recover, helps for the same reason. The next concept, DAgger, attacks the problem directly.