FastMix Optimizes Data Mixtures for Large Models Efficiently.

The performance of large models heavily relies on large and diverse datasets, but identifying the optimal data mixture for both pre-training and post-training remains a significant challenge. Traditional methods often depend on heuristics or resource-intensive simulations. FASTMIX addresses this by introducing a novel framework that automates data mixture discovery while only requiring the training of a single proxy model. This significantly enhances efficiency and scalability compared to previous approaches. The core innovation lies in reformulating mixture selection as a bilevel optimization problem. Under this reformulation, optimizing mixture ratios becomes mathematically equivalent to assigning per-source loss weights under uniform source sampling, embedding mixture coefficients directly into a differentiable optimization objective. FASTMIX employs an approximate iterative procedure, alternating between updating model parameters based on current mixture ratios and updating those ratios based on validation feedback. This gradient-based approach has shown to outperform baseline methods across pre- and post-training tasks, drastically reducing the search cost for optimal data mixtures.