SAGE Improves Spiking Transformer Training with Adaptive Gradients

Spiking Neural Networks (SNNs) are gaining attention as an energy-efficient alternative to traditional deep neural networks, primarily due to their sparse, event-driven computation. However, training SNNs presents a significant challenge because the non-differentiable nature of spike functions necessitates the use of surrogate gradients. The fixed shape of these surrogate gradients can often be suboptimal across different layers and various stages of the training process, hindering performance. To address this, a new mechanism called SAGE (Surrogate-gradient Adaptation via Attention-Guided Entropy) has been developed for Transformer-based SNNs. SAGE introduces an uncertainty-modulated approach that dynamically adjusts the surrogate-gradient slope during training. It achieves this by estimating block-level uncertainty directly from the normalized self-attention entropy within the Transformer architecture. Crucially, SAGE only modifies the training-time surrogate parameter, leaving the inference model completely unchanged. This preserves the original architecture and the low deployment cost that makes SNNs attractive, while significantly enhancing optimization flexibility. Experiments conducted on CIFAR-10 and CIFAR-100 datasets demonstrated that SAGE consistently achieved 1-2% accuracy gains over baselines that use fixed-surrogate gradients across multiple simulation time steps. These results highlight the potential of using attention-derived uncertainty as a lightweight yet effective training signal for adaptive surrogate-gradient learning in energy-efficient Transformer-based SNNs.