New Distillation Method Improves LLM Efficiency and Accuracy

Researchers have identified a significant limitation in current low-rank clone (LRC) distillation techniques for large language models. Existing LRC methods deploy a full-width student MLP but only train a subset of its parameters, leaving a large portion of the deployed matrix's independent linear degrees of freedom untrainable. This means that computational resources are spent on parameters that never contribute to learning. To address this "reachability gap," the paper proposes two new realizations: Dense-LRC and CORE-LRC. These methods modify the training objective to encompass the entire deployed matrix, without altering the deployed model's shape, parameter count, or inference FLOPs. By ensuring that all deployed parameters are trainable, the new approaches recover stranded capacity. Experiments show substantial gains in accuracy and token efficiency across various large language models, including Llama and Qwen. For instance, one realization achieved a 2x token efficiency on Qwen, matching original accuracy at half the tokens. A 1.5B student model, after distillation, matched its 9T-token teacher's performance on a macro-average of 9 tasks.