LLMs for Robot Task Planning

A large language model knows a lot about everyday tasks, so it can suggest sensible steps for "I spilled my drink". But it does not know what this robot can do, or what is in front of it. The SayCan approach multiplies two scores: how useful the language model thinks each skill is, and how likely the robot is to succeed at it here.

Request A sponge is within reach There is an apple on the counter

Combining two scores

For each skill $\pi$ the robot knows, the language model scores how likely that skill is to be a useful next step for the request $\ell$ given the steps so far. A learned value function scores how likely the skill is to succeed from the current state $s$. The robot picks the skill with the best product:

$$\pi^* = \arg\max_{\pi}\ \underbrace{p(\pi \mid \ell, \text{steps so far})}_{\text{useful}} \times \underbrace{p(\text{success} \mid \pi, s)}_{\text{possible}}$$

What to notice

The language scores alone would happily choose "find a sponge" or "bring an apple" even when there is none. The feasibility score vetoes those. Untick "a sponge is within reach" and the robot picks a different way to help. The scores on this page are made up to show the method, not output from a real model. Today, vision-language-action models often fold both steps into one network, but checking feasibility separately remains a useful safety idea.