New Benchmark Evaluates LLM Reasoning for Wet-Lab Protocols

A new benchmark, BenchBench-Protocol, has been introduced to rigorously evaluate the reasoning and modification capabilities of large language models (LLMs) in the context of wet-lab scientific protocols. This benchmark is unique because its 149 tasks are not expert-elicited but are reconstructed from actual modifications scientists made to published protocols during their real experimental work. Adapting a published protocol for a new experiment is a routine yet complex task for wet-lab scientists, requiring careful consideration of prior choices and downstream steps. BenchBench-Protocol captures this complexity by deriving tasks from the differences between original and modified protocols, which then form the basis for queries and weighted rubric elements for correct responses. The benchmark draws from 96 source protocols across nine domains of wet-lab biology, with all tasks highly rated by domain experts. Initial evaluations of nine closed and open models show that Claude Opus 5 achieved the highest score at 59.2%, indicating that the benchmark remains challenging and unsaturated. This development is crucial as LLMs become increasingly integrated into life-sciences research, necessitating robust evaluations for practical wet-lab applications.