New Study Explores Layer Patching for Efficient Model Size Interpolation

Researchers have conducted the first systematic study into layer patching for model size interpolation. This technique allows for the creation of new language models with intermediate sizes and performance levels by combining layers from a smaller "student" model and a larger "teacher" model, bypassing the need for extensive retraining. The process involves replacing blocks of student layers with corresponding teacher layers. The study frames the challenge of selecting the best layer subsets as an optimization problem, demonstrating it can be solved using a shortest-path approach on an acyclic graph. Experiments revealed that the patching strategy significantly impacts interpolation behavior, with simple sequential methods often yielding strong results. A new greedy algorithm, KLPatch, based on KL divergence, was also introduced, showing further improvements over existing methods.