World-Action Models

Instead of predicting actions directly, some robot models first imagine what should happen, like a short video of the task being done, and then work out what actions would produce it. A world-action model generates future observations and actions together. Imagining the outcome first lets the model draw on everything it has learned about how scenes change.

Imagined keyframes Inverse dynamics model accuracy Imagination errors

Imagine, then act

Given the current observation $o_t$ and a goal, a generative model proposes future states $\hat o_{t+1}, \dots, \hat o_{t+k}$. An inverse dynamics model then answers a simpler question for each pair: what action takes the robot from one to the next?

$$\hat a_t = g_\psi(o_t,\ \hat o_{t+1}) \qquad \text{trained on real } (o_t, a_t, o_{t+1}) \text{ triples}$$

Some models predict states and actions jointly in one network, and some, like video-generation policies, condition actions on generated frames.

What to notice

When imagination and inverse dynamics are both good, the executed path follows the imagined one to the goal. If the imagined states are physically impossible, for example drifting through the obstacle, no action can reproduce them, and execution diverges. A weak inverse dynamics model also drifts. More keyframes make each step easier to invert, but give the imagination more chances to go wrong. The approach is attractive because the imagination part can learn from video without action labels.