NEOSTI: A Neuromorphic Electronic-Opto Spatial-Temporal Hybrid Image Sensor

Image sensors in machine vision systems face significant challenges related to energy efficiency and processing capability when storing, transferring, and processing massive amounts of data. In humans, over 80% of information processed by the brain is obtained through the eyes, which are capable of detecting and synchronously processing information with extremely low overall power consumption. Inspired by the biomimetics, here we propose a Neuromorphic Electronic-Opto Spatial Temporal Imager (NEOSTI), the smallest all-in-one eye size fully integrated vision system enabling acquisition and operation in typical indoor/outdoor non-coherent environments, under both natural and artificial lighting conditions, without any extra requirement of the light source, such as laser or coherent light source. NEOSTI combines processing-pre-sensor (PPS) in optical domain, processing-in-sensor (PIS) with non-linear acquisition capability while optical to electronic converting, and processing-near-sensor (PNS) in electronic domain, enabling parallel data computing capabilities while sensing. NEOSTI also integrates a low complexity Binary Neural Network (BNN) on the chip to process image semantic information. It attains near-human performance in five static and dynamic visual processing tasks.