Learned Locomotion with Reinforcement Learning

Today's legged robots often learn to walk in simulation, running thousands of trials in parallel. The danger is that a policy tuned to one exact simulator fails on the real robot, whose mass, friction, and delays are a little different. Domain randomization trains on many slightly different simulators so the policy works across all of them. A tiny balancing policy stands in for a walking policy here.

Randomize mass, friction, and delay during training Randomization range

Reinforcement learning, in brief

The policy here is two numbers, gains on the pole angle and its rate, $u = -k_1\theta - k_2\dot\theta$. Training uses random search, a simple RL method: try small random changes to the gains, keep the ones that earn more reward. Reward is +1 for each step the pole stays up:

$$\mathbf{k} \leftarrow \mathbf{k} + \alpha \sum_i \big(R(\mathbf{k} + \boldsymbol\delta_i) - R(\mathbf{k} - \boldsymbol\delta_i)\big)\,\boldsymbol\delta_i$$

With domain randomization, each trial uses a simulator whose parameters $\boldsymbol\xi$ are drawn at random, so training maximizes $\mathbb{E}_{\boldsymbol\xi}[R]$.

What to notice

Without randomization, training reaches full reward in its one simulator, then fails often on real robots that are heavier or slower to respond. With randomization, training is a little harder, but the policy succeeds on most real robots. Real systems train neural network policies with PPO across thousands of randomized simulations on a GPU, but the reason it transfers is the same.