Q-Learning

Q-learning learns the value of taking each action in each state, called Q, purely from experience, with no model of the world. After every step it nudges its estimate toward the reward it just got plus the best value it expects next. Over many episodes the values settle, and acting greedily on them gives a good policy.

Learning rate α Exploration ε Discount γ Episodes per second

The update

After taking action $a$ in state $s$, receiving reward $r$, and landing in $s'$:

$$Q(s, a) \leftarrow Q(s, a) + \alpha\Big[r + \gamma \max_{a'} Q(s', a') - Q(s, a)\Big]$$

The agent picks the best known action most of the time, but a random one with probability $\varepsilon$, so it keeps exploring.

What to notice

At first the values are all zero and the agent wanders and falls off the cliff often. The values near the goal fill in first, then spread backward along the path, one step at a time, which is how reward propagates. The greedy policy that emerges hugs the cliff, the shortest route, because Q-learning learns the value of the best actions, ignoring that exploration sometimes pushes it over the edge. Its return during training stays noisy for that reason. A larger learning rate learns faster but less steadily.