Vision-Language Models for Robots
A vision-language model, or VLM, takes an image and text and answers in text. Robots use them in several roles: to describe and find objects, to point to where to act, to break a request into steps, and to check whether a task actually worked. They bring broad common sense, but their spatial precision and reliability still need checking.
Roles a VLM can play
A VLM encodes the image into tokens and generates text conditioned on the image and the prompt, \(p(\text{answer} \mid \text{image}, \text{prompt})\). For a robot it can act as a perceiver (what is here, where is it), a pointer (output image coordinates to grasp or place at), a planner (a list of steps), or a success detector (did it work, with a confidence). Pointing outputs are pixel coordinates, which the robot turns into 3D with depth:
What to notice
The answers here are scripted to show each role; a real model's would vary with wording and image. Pointing gives a pixel location the robot can reach for. The success check gives a confidence, which matters, since VLMs can be confidently wrong, especially about small details like whether a drawer is fully shut. In practice robots combine VLM answers with geometric checks, and often ask several times or use several views.