MirrorCraft Benchmarks LLM Agents Under Dynamic Minecraft Rules

The evaluation of large language model (LLM) agents in environments like Minecraft typically occurs under fixed game rules. However, real-world scenarios often involve dynamic or changing conditions, which current benchmarks don't adequately address. This research introduces MirrorCraft, a novel benchmark designed to test LLM agents' adaptability to hidden rule changes in Minecraft. MirrorCraft creates "Mirror" worlds that are exact copies of "Vanilla" worlds, except for specific server-side rule modifications (e.g., altered recipes or item drops). Crucially, terrain, objectives, and interfaces remain consistent between paired worlds, allowing for controlled evaluation of an agent's ability to cope with unexpected rule shifts. The benchmark uses a metric called Rule Intervention Effect (RIE) to quantify performance changes between matched Vanilla and Mirror worlds. Experiments across various biomes, rule suites, and agent configurations revealed that hidden rule changes significantly impact performance. Among agents not explicitly given rule descriptions, the ReAct configuration achieved the highest pooled Mirror score, indicating better adaptability. Providing exact rule descriptions offered only modest gains, suggesting that agents still struggle with dynamic adaptation even with explicit information.