Learnable Wavelet Activations Enhance Continual Learning Plasticity.

Continual learning, where models acquire new tasks sequentially without forgetting previous ones, faces a significant hurdle known as plasticity loss. Existing activation functions often suffer from spectral bias, favoring low-frequency variations, while fully learnable activations can lead to catastrophic forgetting. To overcome these limitations, researchers propose a new approach using learnable wavelet activations. This method explicitly decomposes the activation function into low-frequency and high-frequency components, directly addressing spectral bias. It also incorporates dynamic wavelet injection to adaptively boost plasticity for new tasks. Furthermore, a regularization strategy is employed to ensure the stability of previously learned knowledge. Theoretical guarantees support the hybrid wavelet architecture's efficiency for approximation and the decoupled learning rate's ability to restore network plasticity for high-frequency information. Empirical results demonstrate superior trainability, generalization, and state-of-the-art performance across various continual learning benchmarks.