Computer vision (for robots)

Computer vision is how a robot makes sense of what its camera sees. It turns a flat grid of pixels into "there's a person 3 metres ahead, facing me, walking forward at 1 m/s." It's done a lot more of the heavy lifting in modern robotics than you'd guess.

The five jobs computer vision does for robots

Classification. "What is this thing?" — given a picture, output a label: dog, traffic light, screwdriver.

Detection. "Where is the thing?" — given a picture, output a bounding box around every object of interest plus its label.

Segmentation. "Which pixels are the thing?" — given a picture, output a precise outline of every object, pixel-perfect.

Depth estimation. "How far is the thing?" — given a picture (or two from stereo cameras), output a depth map.

Tracking. "Where is the same thing going?" — given a sequence of pictures, follow each object frame by frame.

Most robot vision pipelines chain these together: detect objects → estimate their depth → track their motion → decide what to do.

How it actually works today

Until ~2012, computer vision was hand-engineered: humans wrote rules ("if the pixels here are dark and edge-aligned, it's probably a road"). It mostly didn't work.

After 2012 (the AlexNet moment), the field switched to convolutional neural networks (CNNs). You feed millions of labelled pictures to a network. The network learns the features on its own.

After 2021, transformer-based vision models (ViT, DINO, SAM) became state of the art. These can be trained on much larger, less-labelled datasets, and they generalise better.

A modern robot like Optimus or Figure 03 has a single neural network that does detection, segmentation, depth, and tracking in one pass — at 30 frames per second, running on a few-hundred-watt onboard computer.

Tesla vs. Waymo — the classic vision debate

Tesla uses vision-only self-driving: just cameras, no lidar, no radar.

Waymo uses sensor fusion: cameras plus lidar plus radar.

Tesla's argument: humans drive with just two cameras (eyes), so cars should be able to. Waymo's argument: cars don't need to be limited to human senses — give them lidar too and they're safer.

Neither approach is finished. The argument is one of the biggest open questions in modern robotics.

---

The first practical use of computer vision most people have touched is the Roomba's navigation cameras. Read How a Roomba decides where to clean.