Efficient Algorithm Boosts Learned Optimizers for Long-Horizon AI Tasks

Learned optimization aims to create superior optimizers using neural networks, surpassing traditional methods like Adam. However, current approaches struggle with scaling to long-horizon problems and consistently outperforming established optimizers. A new algorithm, Efficient Long-hOrizon (ELO) learning, addresses these issues. ELO optimizes meta-training by strategically reallocating computational resources to address longer failure scenarios, making the learning process more efficient. It also incorporates a decoupled progressive expert supervision mechanism, which provides stable learning signals and improves the generalization capabilities of the learned optimizers. Empirical studies show ELO's effectiveness across various tasks, including large language model training (GPT-2) and image classification (ViT, ResNet-50). ELO-trained optimizers consistently outperform well-tuned AdamW and remain competitive with Muon, all while requiring significantly less GPU-hours for meta-training.