Semantic and Instance Segmentation

Segmentation labels each pixel instead of drawing boxes. Semantic segmentation says what class each pixel is, so two mugs share one color. Instance segmentation also separates individual objects, so each mug gets its own mask. Robots need instances to grasp one object at a time.

Mask type Mask edge error

Scoring masks

A predicted mask $M$ is compared with the true mask $G$ by their intersection over union, the same IoU used for boxes but counted in pixels:

$$\operatorname{IoU} = \frac{\lvert M \cap G\rvert}{\lvert M \cup G\rvert}$$

Averaging IoU over classes gives mean IoU, the usual score for semantic segmentation.

What to notice

In semantic mode, the two mugs and the two cans each merge into one region, so a robot could not tell where one mug ends and the next begins. Instance mode splits them. As the edge error grows, small and thin objects lose IoU fastest, since a few wrong pixels at the border are a large share of a small mask. Panoptic segmentation combines both: an instance for every object and a class for every pixel of background.