GPTNT Benchmarks Real-Time Multimodal Agent Collaboration

This paper introduces GPTNT, a novel benchmark designed to assess the real-time collaborative capabilities of multimodal AI agents. The benchmark is built around the cooperative video game "Keep Talking and Nobody Explodes," where two agents must work together to defuse procedurally generated bombs against a live countdown. One agent possesses visual and manipulative access to the bomb but lacks defusal instructions, while the other has the instructions but cannot see or interact with the bomb. Success hinges on effective and efficient communication under conditions of time pressure and information asymmetry. Unlike turn-based evaluations, GPTNT requires agents to communicate asynchronously and act in real time. The benchmark is structured to isolate collaborative performance from memorized solutions, allowing researchers to withhold either the instruction manual or the partner. Experiments with state-of-the-art closed- and open-source models reveal significant challenges: none of the tested models successfully defused a single bomb in real time, a task human players readily accomplish. The study identifies critical weaknesses in areas such as state tracking, efficient action under pressure, handling ambiguous communication, and error recovery. GPTNT is released as a public benchmark to drive advancements in collaborative AI.