LSR-Synth Benchmark Assesses Symbolic Discovery Beyond Memorization

A new study delves into the LSR-Synth benchmark, a tool specifically created to assess scientific equation discovery while actively preventing models from merely recalling known equations from their training data. The benchmark achieves this by introducing novel synthetic terms into established scientific mechanisms and filtering for tasks that are new, solvable, and scientifically plausible. The research explores a specific measurement question: how well do scientific priors provided by language models (LMs) distinguish themselves from traditional operator search methods that do not access task semantics? To investigate this, the authors constructed a semantics-free baseline using a fixed, publicly documented vocabulary and systematically varied candidate coverage through semantic blinding and library weakening. The findings indicate that, under current task conditions and scoring protocols, the fixed vocabulary already covers most tasks. Language model-generated candidates rarely expanded the set of solvable instances unless the vocabulary coverage was deliberately disrupted. While strict out-of-distribution evaluation lowered overall success rates, it did not alter this fundamental relationship. The conclusion is that while LSR-Synth effectively controls against memorization, current tasks are better suited for evaluating the fitting and recombination of unseen expressions rather than uniquely identifying contributions from LM priors beyond a predefined search space.