ToolGate Automates Scientific Benchmark Construction

Creating high-quality scientific benchmarks, especially those requiring computations with specialized software, is a labor-intensive process often involving domain experts manually crafting and cross-checking tasks. While language models can rapidly generate candidate questions, the challenge lies in efficiently accepting or rejecting them. A new system called ToolGate addresses this by providing an executable acceptance pipeline for tool-dependent scientific benchmark construction. ToolGate treats each generated item as a proposal and subjects it to a three-stage validation process. First, a "local-verification gate" ensures that an executable solution script can reproduce the proposed answer when run with the specified scientific software. Second, a "randomized no-tool screening" rejects candidates that a language model can solve without needing the software, thus filtering out trivial questions. Finally, a "tool-using agent" must be able to solve the surviving questions within a fixed time limit. The researchers instantiated ToolGate using FEniCSx, generating 500 initial candidates. The pipeline successfully filtered these down to 128 unique, non-trivial, and tool-dependent questions. This process transforms the traditionally manual and repetitive tasks of answer checking and difficulty screening into an auditable, automated workflow, allowing domain experts to focus on design and final review.