Multi-Agent AI Discovers New Optimizer Programs for Transformers

Designing effective optimizers for modern deep learning, particularly for transformer models, is a complex scientific challenge that requires balancing various factors like optimization geometry, numerical stability, and empirical generalization. Traditional automated discovery methods often either lack structure, leading to invalid programs, or are too constrained, limiting innovation. To overcome these limitations, OPTScientist, a novel multi-agent framework, has been introduced.OPTScientist frames optimizer design as a structured scientific search problem within a typed domain-specific language (DSL). This DSL allows for expressing optimizer updates through modular components such as direction, scaling, and regularization. The framework employs four specialized AI agents—Theorist, Designer, Engineer, and Reviewer—who collaborate in an iterative loop. These agents propose hypotheses, synthesize DSL candidates, compile and evaluate the optimizers, and provide critical feedback.A key feature of OPTScientist is its ability to extend its search space. When repeated failures indicate representational bottlenecks, the framework can propose small extensions to the DSL. This approach led to the discovery of RS-MR, a reduced-state matrix optimizer that demonstrated improved transformer pretraining performance over strong baselines. This research suggests a promising future for automated optimizer science, grounded in theoretical principles, structured programming, and closed-loop experimentation.