Safe Reinforcement Learning

A robot that maximizes only its reward will take any risk that pays. Safe reinforcement learning adds a cost, such as near-misses with people, and a limit on how much cost is acceptable. One common method adjusts a penalty weight automatically: raise it while the robot breaks the limit, lower it once the robot is comfortably safe.

Method Allowed near-misses per hour Penalty learning rate

A constrained problem

The robot chooses its cruising speed $v$. Faster earns more deliveries per hour, $R(v)$, but causes more near-misses, $C(v)$. The goal is the best reward while keeping cost under a limit $d$:

$$\max_v\ R(v) \quad \text{subject to} \quad C(v) \le d$$

The Lagrangian method maximizes $R(v) - \lambda\,(C(v) - d)$ while adjusting the penalty $\lambda$ up when the limit is broken and down otherwise: $\lambda \leftarrow \max(0,\ \lambda + \eta\,(C - d))$.

What to notice

Unconstrained, the robot speeds up until its near-misses are far over the limit. With the adaptive penalty, it settles at the fastest speed that respects the limit, after some overshoot while $\lambda$ catches up, and a faster penalty learning rate reacts sooner but oscillates more. The shield simply blocks any speed known to break the limit, so it is never violated during training, but it needs a trustworthy model of what is unsafe. Real systems often combine both.