Model-Based RL and Learned Dynamics

Model-free RL, like Q-learning and PPO, learns what to do directly from huge numbers of trials. Model-based RL first learns how the world responds to actions, then plans with that learned model. It usually needs far fewer real trials, which matters for real robots, but its plans are only as good as its model.

Real trials used to learn the model Noise in the real robot

Learn a model, then plan

Collect transitions $(s, a, s')$ and fit a dynamics model by regression, here a linear model of the robot's velocity response:

$$\hat s_{t+1} = f_\phi(s_t, a_t), \qquad \phi^* = \arg\min_\phi \sum \lVert s_{t+1} - f_\phi(s_t, a_t)\rVert^2$$

Then plan with the model: here the robot inverts the learned model to find actions that should carry it smoothly to the target, executes them, and replans only every 20 steps, so the quality of the model really shows.

What to notice

With only a handful of trials, the learned model is rough, so its predicted path, dashed, drifts from reality, and the robot stops off target. With 20 or so trials, the model is good enough to reach the target, while a model-free random search needs hundreds of trials to do as well, as the plot shows. Noise makes the model harder to learn. Methods like PETS, Dreamer, and TD-MPC build on this idea, often with neural network models.