A new type of imaging sensor inspired by the functioning of the human brain can detect light and store data at the same time. What’s more, the device can forget the data it no longer needs, paving the way for future breakthroughs in robotic-vision efficiency.

Today’s digital cameras rely on complementary metal-oxide-semiconductors (CMOS) or charge coupled devices (CCDs), which convert photons into electrons. These devices, however, cannot store the images. The data have to be moved into memory and further for processing.

“Conventional camera sensors, which capture images, immediately forget them unless the information is transferred to a separate memory component,” says Larry Cheng, a professor of electrical engineering and computer science at Oregon State University. “Our device can see and remember it, and most importantly, gradually forget it. This kind of gradual forgetting is a key and very important feature of the device.”

Cheng and his colleagues described the device, a type of phototransistor, last month in the journal Advanced Functional Materials. He says that typical AI-driven image-recognition algorithms analyze captured video feed frame by frame to detect moving objects. Instead, the device developed by his team at Oregon State stores the recent history of light intensity that hits it. By doing so, the phototransistor flags changes and patterns of interest. The duration for which it remembers those changes can be modified based on specific needs. For example, a drone flying at 250 kilometers per hour needs only a short trail of changes, while a doorbell camera looking for strange people lingering around needs a longer sequence.

“The ability to tune the memory timescale is a key advantage of our approach, allowing the same sensor to be adapted for different AI vision tasks while improving speed and energy efficiency,” says Cheng.

He adds that the ability to do such basic processing directly on the sensor could pave the way to massive energy-demand reductions. Commercial cameras constantly shuffle data among sensors, storage devices, and processors making it relatively energy intensive to run image-recognition algorithms.

Experimental Oxide Phototransistor

The prototype device built by the researchers is a four- by four-pixel array, about the size of a USB stick. The top of the array is coated with a transparent light-absorbing layer of organic material that transforms the incoming light into electric charge.

Cheng explains that when photons hit the photoactive layer, they produce electrons and create holes. The electrons are transferred into the underlying transistor channel, which is made of indium gallium zinc oxide (IGZO). It’s the holes that provide the basis of the device’s memory function .

“The holes become trapped within isolated organic semiconductor aggregates because of energy barriers in the photoactive layer,” Cheng says. “These trapped holes continue to electrostatically modulate the [transistor] channel even after the light is turned off, allowing the device to retain a memory of recent illumination.”

The amount of charge gradually decays, but by applying voltage to the photoactive layer, the researchers could alter how long it lasts. A positive voltage, Cheng says, pushes the trapped holes further away from the transistor channel, reducing their effect and speeding up their decay. That results in faster forgetting. On the other hand, a negative voltage pulls the holes closer to the transistor channel and slows down the degradation process. As a result of such an intervention, the device retains the memory for hours or more, he says.

“This tunable memory enables the same device to adapt its temporal response to different applications, from tracking fast-changing events to storing longer-term visual information,” he says, adding that the memory feature of the organic photoactive layer was discovered by accident.

The researchers chose an IGZO transistor for its transparency to visible light, which means the transistor doesn’t contribute to light absorption.

“This decouples the electrical transport from the light-sensing function, which is handled by the organic photoactive memory layer,” says Cheng. IGZO is widely used in display technologies because charge travels quickly through it, yet transistors made using the material leak little current; additionally it is compatible with large-area fabrication. Together, the two materials enable each pixel to both detect light and retain a memory of recent illumination within a single device.

The way the device works, Cheng says, is inspired by the functioning of the human brain. The charge in the transistor acts like the neurotransmitter dopamine, which strengthens connectivity between synapses, the links between neurons, and thus the memories we keep.

“Our current work demonstrates the concept at the device level and with simple imaging demonstrations,” he says. “The next step is to scale the technology to larger pixel arrays and develop an integrated imaging prototype to showcase real-time temporal imaging and on-sensor processing. We hope to demonstrate these capabilities in the near future.”