Diffusion Policy

Some tasks have several good answers: go around an obstacle on the left or on the right. A policy trained to predict the average action would split the difference and drive straight into the obstacle. A diffusion policy instead learns to turn random noise into a sample from the whole distribution of good actions, one denoising step at a time, so it commits to one mode.

Denoising steps Samples Show the averaged (regression) answer

Denoising

Training adds noise to good actions $\mathbf{a}_0$ at many noise levels and teaches a network to predict that noise. Sampling runs the process backward: start from pure noise $\mathbf{a}_K$ and repeatedly remove a little of it, each step conditioned on the robot's observation $o$:

$$\mathbf{a}_{k-1} = \alpha_k\Big(\mathbf{a}_k - \gamma_k\,\varepsilon_\theta(o,\ \mathbf{a}_k,\ k)\Big) + \sigma_k\,\mathbf{z}$$

Here the "network" is the exact denoiser for a known two-mode distribution, so you can see the process clearly.

What to notice

Early steps are pure noise. As denoising proceeds, each sample is pulled toward one of the two clean paths and commits to it, above or below, never through the middle. The averaged answer, what a plain regression policy trained with squared error learns, goes straight through the obstacle. With too few steps the paths are still rough. Diffusion policies, introduced by Chi and colleagues in 2023, made this a standard approach for learning robot manipulation from demonstrations.