New RL Fine-Tuning Improves LLM Multi-Task Capability

This research delves into the mechanisms behind reinforcement learning (RL) post-training for large language models (LLMs), which is known to enhance complex reasoning. While previous interpretations often focused on increased and diverse circuit activation, this paper challenges that view by distinguishing between activation and control. It argues that an activated circuit doesn't necessarily control the reward gain achieved during post-training. The authors introduce the "Post-training Control Coefficient" to quantify a component's influence on reward gain, revealing a "Shared Control Bottleneck" where control can become overly concentrated across tasks, even if activations are diverse. The study identifies an "Activation-Control Gap," where highly shared activations can coexist with task-specific control, but a small gap indicates a collapse of control into shared directions, leading to a loss of task specificity. To mitigate this, they propose Control-Diverse Reinforcement Fine-Tuning (CD-RFT). This method regularizes the post-training loss by incorporating the Shared Control Bottleneck, aiming to decouple control and prevent this collapse. Despite the exact regularizer gradient requiring computationally intensive second-order differentiation, the researchers derived a first-order proxy with minimal overhead. Experiments on Qwen2.5-7B demonstrated that CD-RFT achieved the largest control decoupling and significantly improved multi-task capabilities across mathematics, code, and logic, outperforming matched baselines. These findings suggest that the Shared Control Bottleneck is both a diagnostic tool and an effective training regularizer, with control decoupling leading to tangible capability gains that also transfer to other models like Llama-3.2-3B.