DataPrep-Bench Evaluates LLM Training Data Preparation Capabilities

This new research introduces DataPrep-Bench, a novel benchmark designed to evaluate the end-to-end capabilities of large language models (LLMs) and AI agents in preparing training data. Unlike previous methods, DataPrep-Bench focuses on two critical aspects: the LLM's ability to construct supervised training data from raw sources and its capacity to evaluate the quality of candidate datasets before actual model training. The benchmark defines "quality" by the data's utility for downstream model training, rather than superficial textual properties. The benchmark operates across six diverse domains and with various base models, providing a unified framework for assessment. For data construction, methods are scored by fine-tuning a base model on their generated outputs. The researchers also present Data-Construction-Skill, an agent that significantly improves upon existing baselines. For data quality evaluation, scoring functions are assessed by their correlation with actual downstream performance, with the new Distributional Alignment Score (DAS) showing strong cross-model correlation across multiple domains. DataPrep-Bench offers a comprehensive and grounded approach to measuring progress in LLM-driven data preparation, highlighting the co-equal importance of both data construction and quality evaluation for effective model development.