Gefen Optimizer Reduces AdamW Memory Footprint by 8x, Boosts Throughput.

AdamW is a widely adopted optimizer in deep learning, but its memory requirements for storing first and second moment states can be substantial, adding roughly two parameter-sized buffers. This can limit the size of models or batch sizes that can be trained, especially in resource-constrained environments. A new optimizer, Gefen, addresses this challenge by offering a memory-efficient alternative that reduces AdamW's memory footprint by approximately 8x while preserving the same performance levels. Gefen achieves this through two main mechanisms: it automatically shares second-moment estimates across parameter blocks and quantizes the first moment using a learned codebook. This translates to a memory reduction of about 6.5 GiB per billion parameters. The method is theoretically motivated by observations that large mixed Hessian entries constrain squared gradient ratios, suggesting natural candidates for sharing second-moment statistics. Gefen infers block structure from initial squared gradients, requiring no architecture-specific metadata or new hyperparameters. Across various experiments, Gefen consistently achieved the lowest peak optimizer memory among AdamW-like methods while matching AdamW's performance. In distributed training setups like FSDP and DDP, its reduced memory footprint allows for larger microbatches, significantly improving throughput and making it a practical drop-in replacement for training larger models or using bigger batch sizes.