LLM Agents Struggle with Complex Operations Research Tasks

This research introduces ORAgentBench, a novel benchmark designed to rigorously assess the capabilities of large language model (LLM) agents in tackling complex, end-to-end operations research (OR) problems. Unlike previous evaluations that often separate modeling from solving or rely on simplified inputs, ORAgentBench presents 107 human-reviewed tasks within isolated environments, complete with natural-language briefs, multi-file data, and specific submission requirements. Agents are tasked with writing and executing solution code, with their outputs judged on schema validity, constraint feasibility, and objective quality. Experiments conducted with fourteen advanced agent-model configurations indicate that current LLM agents are far from achieving reliable performance in practical OR applications. The most effective agent managed to pass only 35.51% of all tasks and a mere 20.59% of the more difficult ones. Furthermore, many submissions, even if feasible, failed to meet the required quality thresholds. A detailed analysis of failures points to significant strategic deficiencies, including agents overlooking operational rules, formulating brittle problem structures, struggling to construct feasible solutions, and failing to adequately improve solution quality. While OR-specific procedural skills did enhance feasibility for hard tasks, they did not consistently lead to better solution quality or overall pass rates. These findings suggest that future advancements in OR agents must move beyond merely generating plausible optimization code towards ensuring dependable, high-quality operational decision-making.