Offline Reinforcement Learning

Offline RL learns from a dataset collected earlier, with no new trials. That is attractive for robots, since trials are slow and risky. But a learner that cannot try things out can fool itself: it may believe an action it never saw is wonderful, simply because nothing in the data says otherwise. Conservative methods penalize or rule out such actions.

Method How much the logging policy explored

Extrapolation error

Q-learning's update uses $\max_{a'} Q(s', a')$. Offline, some actions $a'$ never appear in the data, so their Q-values are never corrected and keep whatever value they started with or drift to. If that value is too high, the policy picks those actions, and learning builds on the error:

$$Q(s, a) \leftarrow r + \gamma \max_{a'} Q(s', a') \quad \text{with } a' \text{ possibly never seen in } \mathcal{D}$$

Conservative methods, such as batch-constrained Q-learning or CQL, limit the maximum to actions supported by the data, or push down the values of unseen actions.

What to notice

The log here comes from a cautious robot that mostly took the long, safe route. Plain offline Q-learning starts every unseen action at zero, which looks better than the −1 per step it actually sees, so it routes the policy through untried moves, and in the real world it wanders or falls off the cliff. The conservative version only chooses among logged actions and reliably reproduces, then shortens where the data allows, the safe route. More exploration in the log gives both methods more to work with.