Data Selection for LLM Reasoning Depends on Model Capacity

This study investigates how the choice of training data impacts the performance of large language models (LLMs) when fine-tuning for reasoning tasks. Specifically, it examines the role of "likelihood-based" data selection, where responses closer to a student model's current distribution are often assumed to be more effective for supervision. The findings challenge this universal assumption, revealing a nuanced relationship between data difficulty, model size, and training time. The researchers observed a "Fast-Fit / Slow-Gain" pattern. Smaller models (e.g., 1.5B parameters) benefit significantly and rapidly from high-likelihood, easier data, which provides stable early improvements. However, for larger models (e.g., 8B parameters) trained over longer durations, low-likelihood, more challenging data becomes increasingly advantageous. This suggests that larger models are better equipped to learn from more complex examples, whereas smaller models might struggle and fall into repetitive behaviors. The paper provides a theoretical framework for distillation, emphasizing that data difficulty, data span, and student capacity collectively determine knowledge transfer. The key implication is that data selection strategies for reasoning tasks should be dynamically adjusted based on the specific model's capacity and the available computational budget, rather than adhering to a one-size-fits-all preference for high-likelihood data.