Reinforcement Learning: MDPs, Rewards, and Returns

Reinforcement learning frames a task as a Markov decision process: states, actions, a reward after every step, and the chance that an action leads where intended. The robot's goal is to maximize its return, the total reward it collects, with future rewards counted a little less than immediate ones.

Route Discount γ Chance a move slips

The return

After each step $t$ the robot gets a reward $r_t$: here −1 per move, +10 at the goal, and −20 for falling off the cliff, which also sends it back to the start. The return adds them up with a discount $\gamma$:

$$G = r_0 + \gamma\,r_1 + \gamma^2 r_2 + \dots = \sum_{t=0}^{\infty} \gamma^t\,r_t$$

A discount below 1 makes rewards that come sooner worth more.

What to notice

With no slipping, the short route earns more: fewer −1 steps before the goal. Add a small chance of slipping and the short route sometimes falls into the cliff, and the −20 penalty drags its average below the safe route. A lower discount cares less about the distant goal, which makes the long route look worse. Everything a learning agent does next, from Q-learning to PPO, is a way of finding the actions with the highest expected return.