LaPrune Enables Controllable Differentiable Sparsity for Large Models

In sparse models, top-k selection is crucial for determining active components, but traditional hard selection methods hinder gradient flow. While continuous relaxations exist, they often tie mask hardness to the selected mass, limiting fine-grained control. This research presents LaPrune, a new differentiable layer designed to address these limitations. LaPrune offers mathematically exact-budget differentiable sparsity. It controls the normalized second moment of the mask while precisely preserving the selected mass. A key component is the LapSum barrier, which maintains the selection mass, while a normalized second-moment constraint progressively shifts the mask from a dense, equal-mass allocation towards a hard top-k selection at each budget level. The paper provides a population prediction for the saturated fraction, a near-binary limiting law, and a tight worst-case guarantee on the near-zero fraction. Crucially, LaPrune's normalized hardness parameter is invariant to score scale, unlike fixed LapSum temperatures. This innovation provides more stable and controllable sparsity for large-scale models.