Grasp Detection

Instead of modeling every object, a grasp detector looks at a depth image and scores many candidate grasps directly: a center, an angle, and a gripper width. Learned networks such as GG-CNN and Dex-Net do this in a fraction of a second. This page uses a simple hand-made scoring rule to show the idea.

Candidates Score threshold Gripper opening

Scoring a grasp

A parallel-jaw grasp is a center $(x, y)$, an angle $\theta$, and a width $w$. This page's score rewards a grasp whose jaws land just outside an object, on empty space at the same height, with the center on top of the object and the object narrower than the opening:

$$Q = \underbrace{[\text{center on the object}]}_{\text{something to hold}} \times \underbrace{[\text{both jaws in free space}]}_{\text{no collision}} \times \underbrace{e^{-(\text{height difference})}}_{\text{flat contact}}$$

A learned detector replaces this rule with a network trained on many real or simulated grasp attempts, outputting a quality $Q$, an angle, and a width for every pixel.

What to notice

High-scoring grasps line up across the narrow direction of each object, since that is where the jaws fit. Grasps across the long direction score badly, because the object is wider than the opening. Objects packed tightly together leave no room for a jaw, so they get few good grasps until a neighbor is removed. Raising the threshold keeps only the most confident grasps.