CivBench Benchmarks LLM Agents in Long-Horizon, Tool-Mediated Environments

This paper introduces CivBench, an open-source benchmark designed to evaluate language model agents within long-horizon, tool-mediated environments, specifically using the game Civilization VI. The benchmark leverages the Model Context Protocol (MCP), with each episode spanning over 300 turns and involving thousands of tool calls across a vast action space. This setup demands continuous planning, state monitoring, and execution from agents, all under conditions of partial observability. The environment provides 76 MCP tools and a narration layer that translates visual game states into structured text. The researchers utilized CivBench to characterize agent behavior across four different model families in 23 admissible runs, noting that this sample is a pilot study rather than a definitive model ranking. They introduced two interface-level metrics: Proactive Monitoring Rate (PMR), which measures how actively agents query latent strategic state, and RAG@10, which assesses whether commitments stated in structured planning reflections are executed within ten subsequent turns. Across the runs, two consistent behavioral patterns emerged despite a shared playbook protocol. Agents frequently under-monitored strategically relevant state, querying victory progress far less often than guided. Additionally, agents often failed to execute near-term commitments from their own planning reflections, with RAG@10 scores ranging from 48.2% to 65.8%. These deviations occurred despite tool access and explicit instructions, suggesting issues with instruction following rather than capability absence. The environment, scenarios, logs, metrics, and analysis pipeline are openly released.