High-order dynamics in an ultra-adaptive neuromorphic vision device

Abstract

Neuromorphic hardware for artificial general vision intelligence holds the potential to match and surpass biological visual systems by processing complex visual dynamics with high adaptability and efficiency. However, current implementations rely on multiple complementary metal–oxide–semiconductor or neuromorphic elements, leading to significant area and power inefficiencies and system complexity. This is owing to a key challenge that no single electronic device, to our knowledge, has yet been demonstrated that can integrate retina-like and cortex-like spiking and graded neuronal dynamics operable across both optical and electrical domains. Here we report a single ultra-adaptive neuromorphic vision device (IxTyO1–x–y/CuOx/Pd) by ingeniously tailoring its electronic properties, enabling uniquely controlled interface and bulk dynamics by charged particles, including electrons, oxygen ions and vacancies. The device highly amalgamates broadband retinal spiking neuron and non-spiking graded neuron, and cortical synapse and neuron dynamics, with ultralow power consumption. Real-time optoelectronic dynamics is elucidated through in situ scanning transmission electron microscopy and validated by technology computer-aided design simulations. An artificial general vision intelligence system based on homogeneous ultra-adaptive neuromorphic vision device arrays is constructed, adaptively supporting both asynchronous event-driven and synchronous frame-driven paradigms for versatile cognitive imaging demands, with superior power efficiency of up to 67.89 trillion operations per second per watt and area efficiency of up to 3.96 mega operations per second per feature size (MOPS/F2).