LLMs Struggle with Statistical Problem Formulation, New Benchmark Reveals

Large Language Models (LLMs) are increasingly used as assistants in data science, but their ability to handle the crucial initial step of statistical problem formulation has been largely unexamined. This upstream task involves interpreting informal user goals and heterogeneous data to identify the appropriate statistical task and relevant variables. To address this, researchers formalized statistical problem formulation into two subtasks: classification of the statistical problem and identification/role assignment of variables. They then developed StatFormBench, a comprehensive benchmark comprising 1,013 samples from statistics textbooks and data science cases, covering 20 coarse-grained and 85 fine-grained problem categories. Testing 14 LLMs (both open and closed-source) on StatFormBench revealed significant limitations. The best zero-shot models achieved only 72.0% fine-grained classification accuracy and 63.2% variable set overlap. No single model consistently excelled across both subtasks, and advanced prompting strategies offered only marginal improvements. This indicates a substantial gap in LLMs' ability to translate real-world scenarios into structured statistical problems.