Reward Design and Reward Hacking
A reinforcement learning agent does not know what you want, only what you reward. If the reward has a loophole, a good optimizer will find it. A cleaning robot rewarded for each piece of dirt it picks up might learn to dump dirt and pick it up again, forever. Writing rewards that cannot be gamed is one of the hardest parts of RL.
Optimizing the written reward
The agent finds the policy that maximizes the expected discounted sum of the reward exactly as written, here by value iteration over the grid:
If a strange behavior earns more of \(r\) than the intended one, the optimizer prefers the strange one.
What to notice
With "+1 per pickup", there is a spot where the robot can drop dirt and pick it up again. Looping there forever earns more than ever finishing, so the optimal agent loops, and never docks: a reward hack. Rewarding each square cleaned only once, plus a bonus for docking, makes finishing the best plan. Rewarding only the final outcome is hard to game but gives no signal along the way, which makes learning slow. Real examples of reward hacking, from simulated robots exploiting physics bugs to boat-racing agents circling forever, are why reward design and checking learned behavior matter so much.