Microscopy Agent Benchmarks Aid Qualification But Lack Generalization.

Research into agentic control of scientific instruments, particularly microscopes, is rapidly advancing, yet established paradigms for engineering such systems are still emerging. A new study investigates the impact of various design choices—including LLM selection, agent count, delegation rules, and RAG parameters—on the performance of agentic microscope controllers. The goal was to understand not only how agents perform on known tasks but also their ability to generalize to novel, unseen tasks. The study developed a comprehensive benchmark and trace-logging framework, evaluating 105 agent configurations across 53 microscopy tests, involving nearly 2,000 individual runs. Direct comparisons revealed clear differences in latency, token usage, cost, and failure modes among configurations. However, a key finding was that surrogate models trained on agent architecture and test results could not reliably predict an agent's performance on new, unseen tasks. This suggests that while these benchmarks are invaluable for qualification, regression testing, diagnosis, and direct comparisons, the current heterogeneous test suites do not support a global configuration model that guarantees performance across novel scenarios.