Hidden Gauge Controls Feature Specialization in ReLU Networks

This paper explores how feature specialization occurs within overparameterized ReLU networks during training, specifically investigating how certain neurons acquire task-relevant features while others become redundant. The researchers discovered that a seemingly invisible parameter, referred to as a "positive-homogeneous scaling gauge," can deterministically influence which of several initially functionally identical neurons ultimately specializes in a particular feature. Through a tractable Gaussian teacher-student model, the study demonstrates that varying this gauge, even with an identical initial function, leads to distinct feature trajectories and a significant separation in specialization time. Assigning a favorable gauge to one neuron among duplicates ensures it becomes the "owner" of a feature, effectively pruning the functional contribution of others. This effect is attributed to differing mobilities for changing a feature's coefficient and direction, providing insights into the underlying dynamics of neural network learning.