New Optimizer Improves Language Model Training Efficiency and Performance.

Language models operate by translating discrete symbols into continuous programs, with the embedding table and LM-head serving as crucial interfaces. This paper highlights that these interfaces possess a distinct gradient geometry compared to dense hidden weights, a characteristic that can be leveraged for optimization. By exploiting this, researchers have developed Ember, a novel, lightweight optimizer. Ember is specifically designed for embedding and LM-head matrices, offering substantial VRAM savings (O(V + D) versus Adam's O(2VD)) and eliminating the need to shard token table optimizer states. Empirical evidence demonstrates Ember's effective scalability across various batch sizes and parameter counts, leading to improved performance in supervised finetuning, reinforcement learning, and pretraining. The work also suggests that token optimization trajectories are surprisingly simple, challenging conventional views of neural network landscapes, and provides an open-source distributed implementation compatible with existing setups.