Transfer Learning for Smaller Transformers Explored

Training large language models from scratch for every size is computationally intensive. This research explores whether knowledge from a large pre-trained Transformer model can be effectively transferred to initialize a smaller sibling model. The study, focusing on the Pythia family, reveals that while internal representations align strongly across different model sizes, the direct transfer of dense weight parameters is largely ineffective and destructive due to structural incompatibilities in components like rotary embeddings, per-head mechanisms, GELU activations, and LayerNorm. The core finding is that the value of conversion lies in initialization, not direct parameter blending. The researchers decompose the conversion process into two independent levers: least-squares compensation and variance-preserving rescale. When applied in continued pre-training, this combined approach offers significant token efficiency, particularly at lower training budgets, outperforming other subcloning variants. For instance, at 30 million tokens, it substantially beats alternatives, reaching desired quality with fewer computational resources. However, the method's effectiveness has boundaries; at very large donor scales (e.g., 5x the donor size), stacking both levers can lead to over-correction due to ill-conditioning in the compensation solve. This suggests a need for dimension-aware regularization for optimal performance across all scales. The research provides valuable insights into efficient model scaling and transfer learning strategies for Transformers.