DomainPilot Optimizes LLM Fine-Tuning Data Mixtures

The training efficacy of large language models (LLMs) is fundamentally constrained by the quality and composition of their training data. Existing dynamic data scheduling methods often face critical limitations in industrial-scale pretraining and supervised fine-tuning (SFT), such as prohibitive computational costs for large corpora, I/O bottlenecks, or reliance on loss signals that conflate noise, difficulty, and novelty. Researchers introduce DomainPilot, a novel domain-level loss-guided two-stage data mixture optimization framework. DomainPilot continuously monitors token-level domain loss during training, capturing per-domain learning dynamics without halting the data pipeline. This signal informs a coarse optimization stage, which uses scaling laws to fit domain-specific convergence curves and derive a principled prior for mixture adjustment. A subsequent fine optimization stage refines this mixture by modeling cross-domain interaction effects through controlled sweep experiments. The entire mechanism is realized via a patch-based architecture that injects domain-aware loss computation into existing training frameworks (e.g., MindSpeed/Megatron-LM) with minimal code changes. Validation on the Qwen3-1.7B model during SFT showed significant performance improvements across various benchmarks (e.g., +2% on MMLU-Redux, +3.8% on LiveCodeBench v5) without increasing total data volume or training costs, demonstrating the efficacy of domain-level signals for efficient mixture optimization.