TickingCollabBench: New Minecraft Benchmark for Multi-Agent Collaboration.

A new multi-agent benchmark, TickingCollabBench, has been developed using Minecraft to evaluate time-sensitive complementary collaboration tasks. This benchmark is designed to reflect four critical characteristics of real-world collaboration: agent heterogeneity, where agents have different capabilities; mandatory collaboration, requiring agents to work together; dynamic environments that change over time; and strict real-time constraints with inherent failure risks. To facilitate this, the researchers created the TickingCollab framework, which allows for the generation of diverse dynamic environments and abstracts Minecraft's primitive APIs. This abstraction enables declarative YAML task specifications, making it easier to compose complex events and scenarios. Building on this, an automated benchmark generation pipeline was designed, where a Large Language Model (LLM) drafts varied task configurations, and a feasibility verifier filters out invalid ones using approximate constraints. Evaluations conducted using TickingCollabBench demonstrated that LLMs frequently struggle in dynamic environments. Their performance falls significantly short of a global-knowledge oracle, primarily due to high latency in language processing and the inherent difficulty of coordinating under partial observability and agent heterogeneity. This highlights the challenges in developing robust multi-agent AI for complex, time-critical collaborative tasks.