Survey Explores LLMs as Optimizers: Direct, Tool-Augmented, and Tool-Creating

Large Language Models (LLMs) are increasingly being utilized for complex mathematical optimization tasks, often without the end-user's explicit awareness. This survey categorizes the field of "LLM-as-optimizer" into three distinct paradigms, each with its own approach to problem-solving and performance characteristics. The first paradigm, "direct optimization," involves LLMs using iterative prompting and heuristic generation to navigate solution spaces. The second, "tool-augmented optimization," focuses on LLMs translating natural language problems into formal specifications and then orchestrating external, specialized solvers. The most advanced, "tool-creating optimization," involves LLMs discovering and generating reusable algorithms or heuristics, which can then be deployed with minimal subsequent LLM cost. The survey outlines the current performance limits based on existing benchmarks and identifies a significant reasoning gap in current LLM architectures. It also highlights the inherent trade-offs between the future potential of direct optimization, which relies heavily on the LLM's internal reasoning, and the enhanced auditability and reliability offered by tool-augmented optimization, which leverages external, verifiable solvers. The paper suggests that even future, more powerful models might benefit from tool-making for operational efficiency in repetitive problem sets.