Learnijoy
ResearchAI ResearchAI Engineering & DevTools

Adam Optimizer's Edge-of-Stability Explained on Quadratic Loss

Yiman Fong, Heng Yang· August 24, 2026 View original

Key takeaways

  • Adam optimizer exhibits an "edge-of-stability" phenomenon, now dynamically explained.
  • On quadratic loss, Adam shows a restoring tendency towards a stability threshold.
  • The study identifies conditions where this edge-seeking mechanism can break down.
  • Understanding EoS is vital for optimizing deep learning training stability.

Who benefits

AI ResearchDeep Learning DevelopmentMachine Learning PlatformsData Science

Summary

Researchers provide a concrete dynamical explanation for Adam's edge-of-stability (EoS) phenomenon by studying its behavior on a one-dimensional quadratic loss function. They prove Adam exhibits a restoring tendency towards its stability threshold but also identify scenarios where this mechanism breaks down.

The "edge-of-stability" (EoS) phenomenon, where optimizers like Adam operate at the boundary of stability, has been widely observed in deep learning, yet its precise underlying dynamics remain elusive. This research delves into this phenomenon by analyzing uncorrected Adam on a simplified one-dimensional quadratic loss function. This controlled setting allows for isolating the optimizer-induced dynamics that drive EoS, free from the complexities of evolving loss geometry. The study characterizes Adam's dynamics across the parameter space, demonstrating that in broad regimes, the optimizer exhibits a "restoring tendency." This means Adam actively seeks and maintains its position near a specific frozen stability threshold. This finding offers a concrete dynamical explanation for why Adam often operates at the edge of stability. However, the research also identifies specific conditions under which this edge-seeking mechanism can fail. These include the emergence of strictly subcritical periodic orbits and specially tuned trajectories that converge to the optimum while consistently remaining in a supercritical state. These results not only provide a clearer understanding of Adam's EoS but also expose its limitations and the conditions under which its characteristic behavior might deviate.

Why it matters

Understanding the fundamental dynamics of optimizers like Adam is crucial for developing more stable, efficient, and robust training strategies for deep learning models, impacting model performance and reliability.

How to implement this in your domain

  1. 1Review your Adam optimizer configurations, especially learning rates and beta parameters, in light of EoS dynamics.
  2. 2Consider implementing adaptive learning rate schedules that account for the observed stability thresholds.
  3. 3Investigate the impact of EoS on the generalization capabilities of your deep learning models.
  4. 4Explore alternative optimizers or modifications to Adam that might offer more stable convergence properties.

Original post by Yiman Fong, Heng Yang

"arXiv:2608.20638v1 Announce Type: new Abstract: The edge-of-stability (EoS) phenomenon of Adam has been widely observed, while its underlying dynamical mechanism is not yet fully understood. We study uncorrected Adam on a one-dimensional quadratic, a clean setting where constant…"

View on X

Originally posted by Yiman Fong, Heng Yang on X · view source

Want to go deeper?

Turn these trends into skills with Learnijoy's hands-on AI & tech courses.

Explore courses

More in AI Research

AI ResearchAI Engineering & DevTools

Harmony Improves Protein-Ligand Flexible Docking with Torsional Diffusion

Researchers introduce Harmony, a harmonic torsional diffusion framework for flexible protein-ligand docking that explicitly accounts for the periodic geometry of angular variables. This method improves ligand pose accuracy and pocket all-atom reconstruction on benchmarks like PDBBind and enhances the physical validity of generated complexes on PoseBusters.

Maksim Zhdanov, Pavel Strashnov, Vladislav KurenkovAug 24, 2026
AI Engineering & DevToolsAI Research

Multilingual Verifier Bias Impacts RLVR in LLM Mathematical Reasoning

A study reveals that exact-match verifiers in Reinforcement Learning with Verifiable Rewards (RLVR) for Large Language Models (LLMs) exhibit significant language-dependent false-negative reward noise in multilingual mathematical reasoning. This bias, particularly pronounced in Japanese, stems from format and script variations, highlighting a cross-lingual selection bottleneck that impedes effective multilingual LLM training.

Chenyu Zhou, Qiliang Jiang, Xu ZhouAug 24, 2026
AI Engineering & DevToolsAI Research

TriPLU Improves Tiny Language Model Performance with Trilinear Product FFNs

Researchers introduce TriPLU, a Trilinear Product Linear Unit, which replaces gated FFNs in tiny decoder-only language models with a direct degree-3 product branch. This approach achieves better validation loss on character-level TinyStories and lower bits per byte on other datasets under low-learning-rate settings, suggesting benefits for small models in specific low-compute regimes.

He ZhangAug 24, 2026