DLR Boosts Low-Rank LLM Pre-Training with Zero Inference Cost

Pre-training large language models (LLMs) at scale is exceptionally resource-intensive. Low-rank pre-training offers a promising solution by factorizing weight matrices to reduce parameters and computational operations, though it often compromises model quality compared to full-rank training. This research proposes Duplicated Latent Residual (DLR), an innovative plug-in designed to improve low-rank pre-training without incurring any additional cost during inference. DLR works by augmenting the standard low-rank output with a fixed, structured residual during training. Crucially, after training, this residual is mathematically absorbed into the up-projection layer, meaning DLR adds zero learnable parameters, FLOPs, or memory overhead during deployment. Experiments across LLaMA models (60M to 7B parameters) show DLR consistently strengthens low-rank pre-training, particularly for models 130M and above, and the folded checkpoints transfer effectively to supervised fine-tuning.