Low-Rank Training: Subspace Non-Identifiability Impacts Optimizer State

New research challenges a fundamental assumption in memory-efficient low-rank training methods for large language models, such as GaLore. These methods typically project gradients onto a low-rank subspace, assuming this subspace is a slowly evolving entity that can be tracked over time. However, the study reveals that beyond a small, reproducible core, this subspace is largely non-identifiable. The findings indicate that estimates of the top-r subspace, even when computed at the same step from different minibatches, diverge as much as estimates taken many steps apart. This suggests that the apparent rotation observed during subspace refreshes is primarily driven by estimator noise, rather than a genuine drift in the underlying subspace. Only a fraction of the directions are reproducible across minibatches, and simple averaging does not effectively recover the rest. This non-identifiability clarifies why certain patches and optimizer state transport mechanisms are more effective. Specifically, it highlights that the first moment (momentum) transports accurately through rotation, aligning with methods like LDAdam, while blindly carrying the second moment (variance) is suboptimal. The research provides insights into the limitations of low-rank assumptions and offers guidance for developing more robust memory-efficient optimizers.