Nonlinear photonic neuromorphic chips for spiking reinforcement learning

Photonic computing chips have made significant progress in accelerating linear computations, but nonlinear computations are usually implemented in the digital domain, which introduces additional system latency and power consumption, and hinders the implementation of fully functional photonic neural network chips. Here, we propose and fabricate a 16-channel programmable incoherent photonic neuromorphic computing chip by co-designing a simplified Mach–Zehnder interferometer (MZI) mesh and distributed feedback lasers with saturable absorber (DFBs-SA) array using different materials, enabling implementation of both linear and nonlinear spike computations in the optical domain through two separate chips. Furthermore, previous studies mainly focused on supervised learning and simple image classification tasks. Here, we propose a photonic spiking reinforcement learning (RL) architecture for the first, to our knowledge, time, and develop a software–hardware collaborative training-inference framework ( in situ photonic training and hardware-aware fine-tuning) to address the challenge of training spiking RL models. We achieve large-scale, energy-efficient (photonic linear computation: 1.39 TOPS/W, photonic nonlinear computation: 987.65 GOPS/W), and low-latency (on-chip 320 ps) deployment of an entire layer of photonic spiking RL. Two RL benchmarks including the discrete CartPole task and the continuous Pendulum task are demonstrated experimentally based on the spiking proximal policy optimization (PPO) algorithm. The hardware–software collaborative computing reward value converges to 200 (−250) for the CartPole (Pendulum) tasks, respectively, comparable to that of a traditional PPO algorithm. This experimental demonstration addresses the challenge of the absence of large-scale on-chip photonic nonlinear spike computation and spiking RL training difficulty, and presents a high-speed and low-latency photonic spiking RL solution with promising application prospects in fields such as robot control, autonomous driving, and embodied intelligence.