New Geometric Method Predicts Optimal Sequential Learning Order for LLMs

Sequential learning, which involves adapting models across different data sources, is highly sensitive to the order in which these sources are presented. This paper reveals that the local impact of learning order can be predicted using a geometric measure called the Lie-bracket commutator of gradient update fields. This computable quantity provides a pairwise score, indicating whether learning from source A then B, or B then A, is more beneficial for a specific target domain. Building on this, the researchers developed a "Lie-Bracket Tournament" planner. This planner efficiently determines optimal learning schedules for multiple domains by using a shared target-gradient reference and leveraging Hessian symmetry. It calculates Borda/row-sum scores with minimal computational cost, avoiding the need to construct a large matrix of all possible pairwise interactions. Empirical results demonstrate the planner's effectiveness. It achieved high pairwise accuracy (98.1% for instruction-SFT, 98.9% for DPO) and maintained strong performance even with more complex scenarios. For curriculum-scale tasks, it recovered the best schedule in a high percentage of trials and significantly outperformed baseline methods in ranking programming language domains and MMLU subjects. This work redefines sequential learning as a geometric tournament problem, offering a scalable solution for optimizing multi-domain learning schedules.