LionVote Improves Lion Optimizer with Per-Layer Learning Rates

Optimizing deep neural networks often involves setting a global learning rate, which may not be ideal for all layers within a complex architecture. Diagnostics reveal that the Lion optimizer, at a single prescribed learning rate, applies an effectively too-high scale to attention and MLP parameters (2.6-2.8x) and normalization layers (~2x) in models like ViT-Tiny/CIFAR-100, indicating a significant cross-layer disparity. To address this, researchers developed LionVote, a mechanism that enables per-layer learning rate adaptation for the Lion optimizer. Each parameter tensor maintains a "compound level," an integer updated periodically based on two diagnostics: gradient direction stability and momentum health, with validation loss serving as a tiebreaker. LionVote's thresholds are derived from geometric identities, EMA time constants, and noise-floor estimates. This adaptive approach resulted in a 69.7% top-1 accuracy on ViT-Tiny/CIFAR-100, outperforming Lion's 69.0% and AdamW's 68.8%. The value of per-layer adaptation varies with architectural heterogeneity and task, with uniform CNNs still favoring tuned SGD and ViT gains being task-dependent.