Small Models Identify Skill Bottleneck Order for Large Model Training.

Training large language models efficiently often involves identifying and resolving skill bottlenecks. This research introduces a novel approach where smaller proxy models are used not just to identify optimal data mixtures, but also to determine the sequence in which larger models should acquire specific skills. The proposed method, LogFloor, is a closed-loop controller that dynamically directs training towards current bottlenecks, generating a phase-ordered skill resolution trajectory. Experiments with Qwen2.5-1.5B models demonstrated that LogFloor reduced token costs by over 56% on average across various bAbI skill slices. Crucially, these bottleneck resolution paths identified by a 70M parameter scout model were transferable to 12B parameter target models, resulting in significant token savings (30-40%) during training. The effectiveness of this approach highlights that the order of skill acquisition is a transferable curriculum structure, offering a more efficient way to manage data and training for large models.