Policy Gradients and PPO

Policy gradient methods improve a policy directly: try actions, then make the good ones more likely. The update is noisy, and too big a step can wreck a policy that was working. Proximal Policy Optimization, PPO, limits how much each update can change the policy, which makes it the most widely used RL algorithm for training robot controllers.

Algorithm Learning rate PPO clip range ε

The policy gradient

For a policy $\pi_\theta(a)$, here a Gaussian with mean $\mu$ and spread $\sigma$, the gradient of expected reward pushes up the log-probability of each sampled action in proportion to its advantage $A$, how much better than average it did:

$$\nabla_\theta J = \mathbb{E}\big[\nabla_\theta \log \pi_\theta(a)\;A(a)\big]$$

PPO

PPO maximizes a clipped surrogate. With the probability ratio $\rho = \pi_\theta(a) / \pi_{\text{old}}(a)$, it stops rewarding changes once $\rho$ leaves $[1 - \varepsilon, 1 + \varepsilon]$:

$$L^{\text{CLIP}} = \mathbb{E}\Big[\min\big(\rho A,\ \operatorname{clip}(\rho, 1 - \varepsilon, 1 + \varepsilon)\,A\big)\Big]$$

What to notice

At a small learning rate, around 0.1, both methods climb to the tall reward peak. Raise it to about 0.6 and plain policy gradient takes wild steps: its mean jumps back and forth past the peak, so its reward swings. PPO's clipping keeps each update small, so it still climbs steadily. Robot locomotion policies, such as those trained in Isaac Lab, are typically trained with PPO.