New Benchmark Evaluates LLM Agents with Scientific Simulators

Large language model (LLM) agents are increasingly integrated with scientific software, but the true impact of tool access on reliability versus complexity remains unclear. To address this, a new diagnostic benchmark, PHREEQC-MCQ-200, has been developed. This benchmark features 200 multiple-choice questions derived from 21 validated PHREEQC scenarios, requiring agents to interact with a scientific simulator to construct inputs, execute simulations, interpret outputs, and provide answers. The evaluation across various LLM families shows that access to a simulator significantly boosts overall accuracy, confirming that grounded execution is essential for many scientific computation tasks. However, the study also uncovered a critical finding: tool-augmented agents sometimes fail on items they previously answered correctly without tools, indicating hidden regressions that simple average accuracy metrics might obscure. Furthermore, the research emphasizes that the protocol for accessing simulator outputs matters. A structured "table-of-contents" interface can reduce token costs and maintain or improve accuracy for advanced models, but it can degrade performance for mid-tier models that struggle with navigating complex structured outputs. The paper advocates for a more holistic evaluation of scientific agents, including item-level retention, output-access sensitivity, trajectory failures, and identifying where the computational chain breaks.