Transformer Model Transfer Learning: Wiring Outperforms Blending

The common practice in developing transformer model families involves training each model size from scratch. This research explores an alternative: converting a larger, pre-trained model into a smaller sibling. The findings indicate that while representations align strongly across different model sizes, the underlying parameters do not, making direct dense weight projection functionally destructive due to its disruption of critical architectural structures like rotary embeddings and LayerNorm. Instead, the study proposes a "wiring" approach, which decomposes conversion into two independent levers: least-squares compensation and variance-preserving rescale. This method proves to be a token-efficient strategy, particularly beneficial at lower training budgets, where it significantly outperforms other subcloning variants. Although the performance gap narrows at larger budgets, this transfer initialization consistently beats training from scratch, offering substantial gains in efficiency and quality, especially for smaller models.