LLM Agents Struggle with Evolving Tools, New Benchmark Reveals

The research introduces MCPEvol-Bench, a novel benchmark designed to assess how well Large Language Model (LLM) agents adapt to dynamic changes in tool interfaces within Model Context Protocol (MCP) servers. Existing evaluation methods often overlook the continuous evolution of tools, leading to an incomplete understanding of agent capabilities in real-world scenarios. To simulate realistic tool evolution, the researchers developed 11 mutation operators and applied them to 123 MCP servers. They then benchmarked 12 state-of-the-art LLMs, including advanced models like GPT-5.4 and Claude-Sonnet-4-6. The findings indicate that even frontier LLMs struggle significantly when toolsets evolve, experiencing performance drops of over 13% and increased errors in planning and reasoning. This highlights a critical vulnerability in current LLM-driven applications and establishes MCPEvol-Bench as a crucial tool for evaluating agent resilience in changing environments.