Singular Learning Explains Simplicity Bias in Deep Networks.

The optimization process in neural networks is profoundly influenced by critical points, which arise from the model's architectural design. These points, where the Jacobian of the model's parametrization becomes rank-deficient, are central to Singular Learning Theory. This research specifically examines such critical points within deep fully-connected networks that utilize monomial activations. Through the application of tools from polynomial algebra, such as Mason's Theorem, the study demonstrates that for sufficiently high activation degrees, criticality in these networks precisely corresponds to the formation of subnetworks. This means that at these critical parameter configurations, certain neurons become either inactive or redundant. This finding offers a rigorous mathematical perspective on the implicit bias observed in deep neural networks, providing an explanation for why these models tend to converge towards simpler, more parsimonious functions, aligning with the principle of Occam's Razor.