Data-DPO Optimizes Data Selection for LLM Post-Training

Supervised fine-tuning (SFT) of large language models (LLMs) often involves selecting a smaller, effective subset of data from vast candidate pools to reduce training costs while maintaining or improving model performance. However, existing data selection methods frequently treat data value as a static property, overlooking how well the data aligns with the target model's current capabilities and learning trajectory. This research introduces Data-DPO, a novel data selection method specifically designed to be "target model-oriented." Data-DPO addresses the compatibility issue by observing the local training feedback of the target model on different samples through a one-step probing process. It then translates the activation differences observed among these samples into pairwise data preferences. A lightweight reward model is subsequently trained to learn these target-model-aware data preferences. In the final selection phase, Data-DPO synthesizes these learned target model preferences with external quality scores and considerations for marginal diversity. This comprehensive approach constructs a more stable and effective training subset. Experimental evaluations on datasets like Vision-Flan and LLaVA-CoT demonstrate that Data-DPO consistently outperforms existing data selection baselines across various data budgets and even surpasses the performance achieved by training on the full dataset.