New Information-Theoretic Measure for Neural Network Plasticity.

Neural network plasticity, defined as a network's capacity to adapt to new tasks, is crucial for effective continual and transfer learning. Current methods for measuring plasticity, such as effective rank or dead neuron fraction, often lack strong theoretical foundations and show poor correlation with actual performance on novel tasks. Researchers have developed a new information-theoretic measure called "local redundancy." This measure is derived from universal compression theory and quantifies the worst-case redundancy within a local model family, specifically parameters in an infinitesimal neighborhood along gradient directions. It is presented as a principled way to assess plasticity. Although direct computation of local redundancy is complex, the study demonstrates that the expected squared gradient norm on a synthetic memorization task serves as an efficient and computable lower bound. Experiments in continual image classification and time series transfer learning confirm that local redundancy is a better predictor of downstream performance than existing measures, proving useful for selecting optimal pretraining checkpoints.