SCAPE Accelerates LLM Training with Extreme Sparse Communication

The pre-training of Large Language Models (LLMs) is increasingly bottlenecked by communication costs, especially in distributed data-parallel and sharded training setups. Existing methods to reduce communication, such as raw gradient sparsification or quantization, often face stability issues with modern optimizers like Adam or have inherent limitations in savings. This new research introduces SCAPE, a novel communication-efficient distributed optimizer designed to overcome these challenges. SCAPE leverages the stability of AdamS's first-moment statistics to enable highly aggressive sparsification of gradients, up to 99%, without degrading LLM quality. Instead of using raw gradients for mask construction, SCAPE generates masks from these more stable first-moment statistics. The system further optimizes by partitioning mask generation across workers to align with optimizer sharding and by delaying mask usage by one step, allowing mask synchronization to overlap efficiently with computation. Additionally, SCAPE reconstructs the necessary quantities for second-moment updates from a single synchronized sparse buffer, eliminating the need for an extra collective communication step. Implemented in Megatron-LM and evaluated on GPT-345M and Llama-500M, SCAPE maintained training stability, validation loss, and downstream task accuracy. For Llama-500M, it reduced end-to-end pre-training wall-clock time by up to 43.3%, and for Llama-1.8B, it achieved up to a 3.26x speedup per step compared to dense AdamS, demonstrating significant efficiency gains.