Hybrid LLM Optimizer Decouples Structure and Parameters

This research addresses a key inefficiency in using large language models (LLMs) for evolutionary algorithms and optimization tasks. Traditionally, LLMs act as a single operator, attempting to simultaneously update both the high-level structural components (like control flow in code) and the fine-grained continuous numerical parameters within a system. This often leads to inefficient trial-and-error loops, as LLMs are not inherently optimized for precise numerical tuning. The proposed solution is a "hybrid nested search" framework. It features an outer loop where an LLM generates a structural blueprint or "sketch" with placeholders for numerical values. An inner loop then employs a dedicated numerical optimizer (such as CMA-ES, gradient-based methods, or MCMC samplers) to efficiently and precisely tune these continuous parameters within the given structure. This modular approach allows for flexible combinations of text-based and numerical optimizers. The framework was validated across diverse scientific domains, including meta-optimizers, code-based policies, and Bayesian inference, consistently demonstrating superior performance compared to both standalone LLM search and pure numerical optimization methods.