Scalable Training for Continuous-Time Spiking Neural Networks

Training deep continuous-time Spiking Neural Networks (SNNs) has been severely limited by the substantial memory required for precise spike-time computation. This process involves evaluating and retaining candidate firing times across intervals determined by presynaptic spike ordering, leading to high memory consumption. The new research proposes Differentiable Spike-Time Discretization (DSTD), a memory-efficient training framework for leaky integrate-and-fire neurons. DSTD converts irregular presynaptic spikes into differentiable weighted events at fixed time points, effectively replacing the input-dependent candidate dimension with a fixed number of time intervals. This approach accurately approximates continuous-time membrane-potential dynamics while drastically reducing activation memory. In dense LIF layers, DSTD achieved up to a 100-fold reduction in peak memory consumption and up to a 20-fold speedup in training time compared to exact spike-time computation. Coupled with synfire-chain-inspired temporal regularization to mitigate dead-neuron failures, these methods enabled the training of complex SNNs (9-layer on CIFAR-10, 20-layer on Fashion-MNIST) on a single GPU, opening new avenues for scalable neuromorphic computing.