Early Token Loss Predicts LLM Reasoning Quality for Efficient Data Curation

This research explores a more efficient method for curating high-quality data for supervised fine-tuning (SFT) of Large Language Models (LLMs), specifically for eliciting strong reasoning capabilities. Current SFT data curation techniques often rely on powerful reasoning models to filter examples based on diversity and difficulty, a process that is both costly and can lead to suboptimal data quality. The paper proposes a novel approach that identifies diverse and challenging reasoning examples by analyzing only the initial reasoning tokens. The core finding is that difficult problems can be reliably detected by evaluating the loss of the first 100 reasoning tokens at a randomly perturbed checkpoint of a pretrained model. Furthermore, the study shows that examples exhibiting similar loss patterns over their first 1,000 reasoning tokens across a small number of perturbed checkpoints, which extrapolate along the fine-tuning trajectory, provably induce similar gradients. This insight allows for a more targeted and efficient selection of training data. The effectiveness of this method was validated through extensive experiments involving the fine-tuning of Qwen2.5-7B and Llama3.1-8B models on the M23K medical reasoning and OpenThoughts-Math datasets. The proposed approach consistently outperformed existing baselines by up to 1.7% in reasoning performance while achieving a remarkable 91% improvement in token efficiency. This signifies a substantial advancement in making high-quality reasoning data curation more accessible and less resource-intensive.