Evaluating Robot Policies

Robot trials are slow and expensive, so evaluations often use 10 or 20 attempts per policy. With so few, a 70 percent policy can easily score 85 or 55. Before claiming one policy beats another, you need to know how uncertain each estimate is. Confidence intervals make that uncertainty visible.

Policy A true success rate Policy B true success rate Trials per policy

Confidence intervals for a success rate

With $k$ successes in $n$ trials, the estimate is $\hat p = k/n$. The Wilson interval gives a 95 percent range that behaves well even for small $n$:

$$\frac{\hat p + \frac{z^2}{2n} \pm z\sqrt{\frac{\hat p(1-\hat p)}{n} + \frac{z^2}{4n^2}}}{1 + \frac{z^2}{n}}, \qquad z = 1.96$$

Its width shrinks only like $1/\sqrt{n}$: four times the trials to halve the uncertainty.

What to notice

With 20 trials each, a 70 percent and an 80 percent policy have heavily overlapping intervals, and in many repeats the worse policy even scores higher. Telling a 10 point difference apart reliably takes on the order of 150 to 300 trials each. Good robot evaluations also test on held-out scenes and objects, keep conditions identical between policies, and report the number of trials along with the success rate.