REGEN Distills LLM Knowledge Cost-Effectively with Offline RL

Researchers have introduced REGEN (Replay-recycling for Expert-to-Generalist Distillation with Offline Reinforcement Learning), a novel approach designed to make the distillation of knowledge from specialized expert LLMs to generalist models far more computationally efficient. Current large-scale online reinforcement learning (RL) and multi-teacher on-policy distillation (MOPD) methods, while effective, are costly and infrastructure-intensive. REGEN addresses these challenges by completely decoupling the rollout sampling process from the backward training process. Instead of requiring multiple teacher models for distillation, it simply reuses the replay memory—a byproduct of the teachers' specialized RL training—and applies offline RL algorithms. This innovative method matches the accuracy of MOPD across diverse tasks such as mathematical reasoning and code generation, but at a substantially lower computational cost, potentially transforming online RL into a data synthesis process rather than a one-off learning stage.