New Benchmark Tests Vision-Language Models' Strategic Reasoning in RTS Games

Researchers have developed RTSGameBench, a novel benchmark aimed at assessing the strategic reasoning capabilities of Vision-Language Models (VLMs). This benchmark utilizes the complex real-time strategy (RTS) game "Beyond All Reason" to challenge VLMs with scenarios requiring coordination, adaptation to opponents, and long-term planning under partial information. RTSGameBench addresses limitations of existing benchmarks by providing a wider range of gameplay, including various matchup structures. It also features diagnostic mini-games to pinpoint specific strategic competencies and an innovative self-evolving generation framework that can create new mini-games from free-form queries. Initial evaluations with state-of-the-art VLMs show that they struggle with tasks demanding tight coordination, multi-agent collaboration, and increased task scale.