RetailBench Evaluates LLM Agents in Long-Horizon Retail Scenarios

Researchers have introduced RetailBench, a novel simulation benchmark designed to evaluate the long-term reasoning and decision-making capabilities of large language model (LLM) agents. Unlike many existing benchmarks that focus on short, well-defined tasks, RetailBench simulates a single-store supermarket operation over thousands of days, presenting agents with complex challenges such as managing pricing, inventory, supplier selection, and customer feedback under cash-flow constraints. The benchmark models retail management as a partially observable decision process, requiring agents to use tools and adapt to dynamic environments. Initial evaluations of seven contemporary LLMs within various agent frameworks revealed substantial performance disparities. Only a few models could complete the 180-day evaluation horizon, and even the best LLM agents lagged significantly behind an oracle policy in terms of net worth and sales. Analysis suggests these shortcomings stem from incomplete information gathering, superficial decision-making, and a lack of consistent long-term strategic planning.