LLMs Generate Superior Multi-Objective Bayesian Optimization Algorithms

Designing effective multi-objective Bayesian optimization (MOBO) algorithms is complex, requiring expert knowledge to balance numerous interdependent design choices. This research introduces an extension of the LLaMEA framework, which leverages large language models (LLMs) as mutation and crossover operators within evolutionary strategies. This allows for the automated generation of complete MOBO algorithm implementations. The framework integrates SMAC hyperparameter optimization directly into the evolutionary loop, enabling the system to discover highly optimized algorithm designs. Across multiple evolutionary runs, approximately 900 algorithms were generated and rigorously benchmarked on a suite of twelve synthetic problems and three real-world engineering challenges. The results are compelling: the strongest generated algorithm achieved superior mean normalized hypervolume on synthetic problems (0.971 vs. 0.869 for qParEGO, a state-of-the-art baseline) while requiring roughly 60 times less wall-clock time. On real-world engineering problems, a generated algorithm also outperformed qParEGO, achieving a mean normalized hypervolume of 0.985 versus 0.971, at approximately 3.4 times lower cost. This demonstrates that LLM-driven evolutionary search can discover highly efficient and effective algorithm designs that are difficult to achieve through traditional manual methods.