ShortOPD Recovers Pruned LLMs for Better Generation

Structured pruning is a hardware-friendly technique to compress Large Language Models (LLMs), but pruned models often perform poorly on free-form generation tasks despite good multiple-choice scores. This gap arises because useful generations are demoted rather than erased, and failures often involve repetitive suffixes. The researchers propose ShortOPD, a novel short-to-long On-Policy Distillation (OPD) schedule to address this. ShortOPD reuses the unpruned model as a teacher to provide dense token-level supervision to the compressed model. Instead of long, wasteful rollouts, ShortOPD detects and prunes repetitive suffixes confirmed by the teacher, focusing training budget on the informative prefixes. This approach significantly improves the compressed model's generation quality across math, code, and open-ended tasks, achieving up to 9 times its unrecovered value and outperforming standard recovery methods. ShortOPD also drastically reduces training time and token usage, making the recovery process much more efficient and practical for deployment-ready LLMs.