LLMs Automate Feature Design for Learning-to-Optimize Methods

Learning-to-Optimize (L2O) techniques aim to accelerate complex optimization problems by training models to predict solutions or guide solvers. A crucial but often overlooked aspect of these methods is the design of feature functions, which map problem instances into inputs for machine learning models. Traditionally, these features are manually crafted and remain fixed, limiting adaptability across different domains. Researchers have developed FunL2O, a unified framework that leverages large language models (LLMs) to automate this feature design process. Operating in a FunSearch-style loop, an LLM proposes executable feature functions. These functions are then evaluated by retraining the L2O model and measuring its performance on downstream optimization tasks. Evaluations on linear, quadratic, and mixed-integer programming tasks, including solution prediction and warm-starting, showed that the LLM-evolved features consistently surpassed traditional hand-crafted representations. This establishes LLM-driven feature evolution as a general and effective approach for automating representation design in L2O, enhancing the efficiency and applicability of optimization methods.