New Research Optimizes LLM Pre-Training with Adaptive Objective Routing

Off-policy distillation is a common technique in pre-training large language models, but the interplay between training data, objective functions, and model capabilities is not well understood. This research delves into top-k-truncated, temperature-scaled off-policy distillation, analyzing how training objectives shape token-level supervision and downstream performance, and how data heterogeneity should guide objective routing. The study demonstrates that the standard language modeling objective and the knowledge distillation objective lead to systematically different capability profiles. This divergence stems from a gradient-level tension between direct reinforcement from observed tokens and alternative supervision provided by a teacher model. Diagnostic metrics were introduced to quantify this tension, showing how support size and distillation temperature influence the trade-offs. Furthermore, the research explores adaptive objective routing, finding that applying different objectives based on data domains (e.g., language modeling for math/code, knowledge distillation for general data) yields better results than single-objective baselines. This suggests that effective pre-training is a data-conditional supervision-design problem rather than a global hyperparameter choice.