Sensing and Perception

The Perception Problem

At 9:58 p.m. on March 18, 2018, a modified Volvo XC90 operated by Uber Advanced Technologies Group was driving 43 mph along North Mill Avenue in Tempe, Arizona. Its LIDAR detected Elaine Herzberg 5.6 seconds before impact. The perception system classified her first as an unknown object, then a vehicle, then a bicycle, cycling through labels without ever recognizing a pedestrian. Each reclassification reset the motion prediction, and the car struck and killed her.

The National Transportation Safety Board (NTSB) investigation (HWY18MH010) identified the classification loop as a central technical failure. The sensors worked. The compute worked. What broke was the gap between pixels and meaning, the place where a cluster of returns becomes a person walking a bicycle across a road.