Low-Precision AI Training Can Silently Halt Learning

Training AI models using reduced floating-point precision can lead to a critical, often unnoticed, issue where learning effectively ceases. This "silent freeze" occurs when the weight updates from gradient descent become so minuscule that they round away, falling below half the unit in the last place (ULP) of the weight's precision. Consequently, the affected weight coordinate stops updating, even if its gradient is non-zero. The study demonstrates that this freeze is deterministic and can be accurately predicted. By analyzing a high-precision training trajectory and the target mantissa length, researchers can forecast when and where these freezes will occur, without needing low-precision data. For instance, a small GPT model trained with bf16-equivalent weights froze permanently mid-run, precisely as predicted. In a 124-million-parameter GPT-2 transformer using 8-bit floating-point weights, dense weights froze at initialization, leading to plateaued validation loss while full-precision training continued to improve. The research also shows that stochastic rounding can prevent this persistent freeze, and its effect is also predictable. This phenomenon is not random noise but a computable aspect of low-precision training, observed across various network types and precision levels.