Multimodal LLMs Struggle with Complex Executive Decisions, Study Finds.

Researchers introduced C-SUITEBENCH, a novel multimodal benchmark designed to assess the decision-making capabilities of large language models in executive business scenarios. This benchmark places nine leading frontier models in the role of a Chief Executive Officer, evaluating their performance across five decision tasks under both text-only and multimodal conditions. The study involved 50 distinct business scenarios to thoroughly test the models' abilities. The findings indicate that incorporating multimodal inputs consistently enhances evidence-centric reasoning, particularly in areas like risk forecasting and justifying decisions to a board. However, a significant "multimodal integration paradox" was uncovered: despite improved visual grounding, adding visual business information actually worsened the models' ability to handle constrained resource allocation tasks. This degradation is attributed to signal crowding, where the combination of multiple visual channels disrupts the models' capacity for constraint satisfaction during the decoding process. This research underscores that visual perception and the ability to act under constraints are separate challenges for multimodal AI agents. It suggests that simply augmenting models with more visual data can be detrimental in high-stakes decision-making contexts, advocating for more selective grounding strategies in the development of future executive AI systems.