DynamicMCPBench Evaluates LLM Agents on Live Servers.

A new benchmark framework, DynamicMCPBench, has been introduced to evaluate Large Language Model (LLM) agents operating over live Model Context Protocol (MCP) servers. Unlike traditional benchmarks that rely on final answers or fixed "ground-truth" tool lists, which can be fragile with live, stateful data, DynamicMCPBench offers a reusable framework. Practitioners can deploy it on their own MCP servers to test models on specific tasks or use it to automatically collect servers and measure an agent's general tool-use capabilities. The framework generates realistic goals, records successful trajectories live, distills these into path-agnostic effect checkpoints, and scores agents on their ability to reproduce these effects, rather than just the final outcome. A large-scale run involving 24 models, 121 servers, and 750 tasks across 15 categories revealed significant challenges for current agents. Even the strongest agents solved only about half of the tasks, and 31% of tasks were not solved by any model. Accuracy sharply declined as the required tool chain lengthened, dropping from 39% on the shortest chains to 13% on the longest. A human validation study confirmed the reliability of the automatic scoring. DynamicMCPBench thus provides a crucial tool for practitioners and exposes a consistent weakness in current agents when handling complex, multi-step agentic tasks.