New Benchmark Evaluates LLMs as CEOs in Strategic Resource Reallocation

Current evaluations of large language models (LLMs) often focus on isolated cognitive tasks, overlooking the complexities of real-world executive decision-making. A key challenge for CEOs is integrating conflicting recommendations from specialized stakeholders, managing information asymmetry, and navigating organizational constraints and temporal dependencies. To address this, a new multi-agent benchmark called CEO-Bench has been developed. This benchmark assesses LLMs on their ability to perform strategic resource reallocation within a simulated, constraint-rich organizational environment over multiple rounds. In CEO-Bench, an LLM agent acts as a CEO, receiving diverse and often conflicting advice from four role-conditioned C-suite advisors (CFO, CTO, COO, CMO), each with their own private information and priorities. The LLM's resulting allocation plan is evaluated across four dimensions: how well it integrates different roles' perspectives, its conditional boldness, its history-sensitive judgment, and the overall validity of the plan. Experiments with five leading models across 13 scenarios showed high structural validity in their plans, but significant divergence in strategic calibration, which proved to be the most difficult aspect. Common failure modes included being overly influenced by a single advisor, defaulting to conservative actions under uncertainty, and failing to recall past decisions. The research also identified a trade-off: models that engaged more deeply with conflicting advice tended to produce less decisive actions. These findings highlight the current limitations of LLMs in complex organizational decision-making and offer insights for future AI-assisted executive systems.