Semantic segmentation

Imagine colouring a photograph with felt-tip pens, but instead of colouring for fun, you colour each object a specific shade: every road surface gets green, every pedestrian gets orange, every parked car gets blue, every patch of sky gets yellow. When you are done, every single pixel on the page has been assigned to a category. That is — almost exactly — what semantic segmentation does.

Semantic segmentation is a computer-vision technique that takes an image and outputs another image of the same size, where each pixel is labelled with the category of whatever it belongs to.

The difference from object detection

Object detection draws a rectangle around each object and labels the rectangle. It is fast and good enough for counting things. But rectangles are crude — they include background pixels, they blur together when two objects are close.

Semantic segmentation is pixel-precise. It does not draw a box around a pedestrian; it traces the exact outline of their coat, right down to the gap between their fingers if the resolution is high enough. The tradeoff is that it is computationally heavier, and it treats every pixel of a given class as identical — two separate pedestrians get the same label, "person," with no distinction between them. (The variant that separates individual instances is called instance segmentation, which combines the best of both worlds at even greater cost.)

A real example

Autonomous vehicles use semantic segmentation on their camera feeds to understand the road surface in fine detail. The model must know, pixel by pixel, where the driveable lane ends and the kerb begins, where a pothole interrupts the surface, and where a white line marks a crossing. A bounding box around "road" would be useless for this level of navigation precision.

The Cityscapes dataset, captured from a camera-equipped car driving through European cities, has been the standard training benchmark for urban segmentation since 2016.

Why it matters for robotics

A surgical robot using semantic segmentation can distinguish a tumour from healthy tissue in a live endoscope feed, pixel by pixel. A farm robot can identify which leaves are diseased and which are healthy without cutting an entire plant. A warehouse robot can find the exact boundary of a product surface to plan a precise grasp.

The richer the pixel-level understanding of a scene, the more precisely a robot can act within it.

When the same pixel-labelling idea is applied to 3D point clouds rather than flat images, robots can segment the entire three-dimensional world around them.