Sim-to-Real and Domain Randomization

Training in simulation is fast, safe, and cheap, but no simulator matches reality exactly. A policy that is perfect in its one simulator can rely on details that are wrong. Domain randomization trains across many simulators, each with slightly different physics, so the real world looks like just one more variation.

Train with randomized friction Real-world friction (relative to the simulator) Randomization range

Training on many simulators

Let $\boldsymbol\xi$ be the simulator's physical parameters, here the friction $\mu$ between gripper and object. Ordinary training tunes the policy for one value. Domain randomization tunes it for a whole distribution:

$$\pi^* = \arg\max_\pi\ \mathbb{E}_{\boldsymbol\xi \sim p(\boldsymbol\xi)}\Big[R(\pi;\ \boldsymbol\xi)\Big]$$

The task: squeeze with force $F$. Two fingers hold the object if $2\mu F \ge m g$, and a force above $F_{\text{crush}}$ breaks it. The reward is +1 for holding without breaking, minus a small cost per newton, so gentler grips score slightly higher.

What to notice

Trained on one simulator, the policy finds the gentlest force that just holds at the nominal friction. On a real object that is a bit more slippery, it drops it. The randomized policy squeezes firmer, enough for the most slippery objects it trained on, while staying below the crushing force, so it succeeds across a much wider range. Randomize too widely and no single force can both hold the slipperiest object and spare the most fragile. This idea, popularized by OpenAI's robot hand and many locomotion results, is a main reason simulation-trained robots now work in the real world.