Learning from Human Preferences

Writing a reward by hand is hard, but people find it easy to say which of two behaviors they prefer. Preference learning turns those comparisons into a reward model, which a robot can then optimize. This is the same idea behind reinforcement learning from human feedback, RLHF, used to train large language models.

Automatic answers are sometimes wrong

The Bradley-Terry model

Each path $\tau$ has features $\boldsymbol\phi(\tau)$: its length, how close it passes to people, and how much it turns. The reward model is linear, $r(\tau) = \mathbf{w}^\top\boldsymbol\phi(\tau)$, and the chance a person prefers A over B is

$$P(A \succ B) = \frac{e^{\,r(A)}}{e^{\,r(A)} + e^{\,r(B)}} = \sigma\big(r(A) - r(B)\big)$$

Fitting $\mathbf{w}$ is logistic regression on the comparisons.

What to notice

Choose paths yourself, or let the automatic answerer, who secretly cares most about keeping away from people, then about length, and little about turning, answer for you. After a dozen or so comparisons, the learned weights point the same way as the hidden ones, even with some wrong answers. Each comparison carries little information, so many are needed, and choosing informative pairs, ones the model is unsure about, speeds things up.