Curved Weight Space Reparameterization Boosts Neural Network Optimization.

Researchers have proposed a new technique called ExLinear for reparameterizing neural network weights, aiming to improve optimization efficiency. The core idea addresses the issue that standard additive updates in optimizers like Adam can lead to disproportionate relative changes for weights of varying magnitudes. ExLinear introduces a curved parameter-space geometry by combining a sign-aware symmetric-exponential pathway with a linear pathway. This design allows additive updates in a transformed logarithmic space to translate into magnitude-proportional changes in the actual weight space, which is more aligned with the multiplicative nature of many neural network operations. The method also includes a "mismatched initialization" strategy that further enhances early optimization. Empirical results on transformer models, particularly larger ones, show that ExLinear significantly reduces the number of training steps required to reach a matched validation loss, demonstrating substantial gains in optimization speed.