Researchers at the University of Arizona have advanced 3D-sensing technology. 

Navigating a chaotic city street during rush hour is a subconscious breeze for humans. Our eyes adjust instantly to glare, shadows, and varying surfaces. For machines, however, it is a nightmare.

Self-driving cars and surgical robots routinely become blinded by mixed-reflectivity surfaces. These technologies get utterly confused by the transition from a matte brick wall to a shiny metallic bumper, or from dull tissue to glistening bodily fluids. Current 3D sensors are usually built to read one or the other, and fail when forced to look at both simultaneously.

The team’s new approach allows sensors to capture images faster and in sharper detail, without getting blinded by tricky, reflective surfaces, using a laser scanner and an event camera.

“Humans already have a built-in 3D camera system – the stereo vision of our two eyes,” said Florian Willomitzer, an associate professor at the U of A Wyant College of Optical Sciences.

“One of our goals is to enable computers and machines to see in 3D better than any human, which is crucial for a multitude of technological challenges, such as reliable navigation of self-driving cars, accurate guidance during robotic surgery or improved sensing capabilities in industrial inspection and biomedical imaging,”  Willomitzer said.

Eliminating massive hardware

Typically, measuring the exact shape of a highly reflective object requires deflectometry. This method projects a known geometric pattern onto a shiny object and reconstructs its 3D shape by analyzing how the pattern deforms across the glossy surface.

There is a massive catch, though. To measure anything complex, the screen projecting the patterns must be enormous. Automotive manufacturers usually build tunnel-like structures lined with screens large enough to inspect a freshly painted car chassis using deflectometry. It is static, wildly expensive, and entirely impractical for a robot navigating a dynamic room.

The Arizona team found a simple way to burn down the hardware requirements. Instead of building a massive screen to project light onto a shiny object, why not turn the room itself into the screen?

“We can use a laser scanner to capture everything in the room, with whatever is inside, including objects with specular, glossy, and matte surfaces, as well as matte walls. We then use our algorithms to separate the diffuse from the specular surfaces and can eventually use all measured diffuse scene parts as a virtual screen for the deflectometry measurement of the specular parts,” explained Aniket Dashpute, the study’s first author. 

Speed via neuromorphic cameras

Mapping a room is fine for a static lab environment, but it does not solve the issue of a speeding autonomous car or a moving surgical tool. To make this practical, the team threw out conventional cameras.

Standard cameras capture a scene frame by frame, like a flipbook.

Researchers integrated a neuromorphic event camera that tracks only changes in local brightness at ultra-high time resolutions. This elimination of redundant data enables the technology to easily capture high-speed, 3D video of moving objects, even in challenging environments with variations in lighting and surface reflectivity.

The prototype system achieves motion-robust 3D tracking of mixed-reflectivity scenes at incredibly high frame rates.

Right now, the technology is confined to a tabletop setup in a University of Arizona laboratory. However, the architecture is fundamentally scalable.

The researchers envision its flexible architecture being adapted for a wide spectrum of 3D imaging applications, ranging from tracking microscopic blood vessels during delicate surgeries to digitally mapping entire rooms and buildings.

The study was published in Nature Communications.