Vision Foundation Models and Open-Vocabulary Perception

Classic detectors only know the classes they were trained on. Vision foundation models are trained on huge sets of images and captions, so they place images and text in the same feature space. A robot can then find "the blue mug" or "something to drink from" without ever being trained on that exact phrase.

Type a query Or pick one Match threshold

Shared feature space

A model like CLIP has an image encoder $f$ and a text encoder $g$, trained so that matching images and captions point the same way. To find an object, compare the query's text feature with each object crop's image feature by cosine similarity:

$$\operatorname{sim}(\text{crop}, \text{query}) = \frac{f(\text{crop})^\top g(\text{query})}{\lVert f(\text{crop})\rVert\,\lVert g(\text{query})\rVert}$$

What to notice

Here the features are a small hand-made stand-in with seven meanings, such as red, blue, food, container, soft, sharp, and drink. Real models learn hundreds of dimensions on their own. Notice that the query "something to drink from" matches the mug and the cup, even though neither label contains those words, because their features overlap with "drink" and "container". This is how open-vocabulary detectors such as OWL-ViT and Grounding DINO work, and DINO-style features add good part-level matching for grasping.