Offline RL Losses Show Distinct Weight-Space Geometries and Performance

Offline reinforcement learning (RL) losses are widely used to distill complex reasoning capabilities from large teacher models into smaller student models. While these methods are typically compared based on their downstream accuracy, this research delves into whether they are mechanistically distinct or if they converge to similar weight updates within the neural network's parameter space. The study trained six different offline RL methods (SFT, RFT, DFT, RIFT, Offline GRPO, DPO) on identical mathematical reasoning rollouts from a Qwen3-4B base model. By analyzing the resulting weight deltas using metrics like cosine similarity, principal-angle subspace analysis, linear mode connectivity, and CKA, distinct patterns emerged. SFT, RFT, and RIFT showed nearly colinear weight deltas and comparable accuracy on GSM8K. DFT diverged more in direction, while Offline GRPO added a substantial orthogonal component to the SFT direction. Crucially, DPO (Direct Preference Optimization) occupied a near-orthogonal subspace, exhibited a mode-connectivity barrier, and showed a collapse in late-layer CKA. Despite using a 10x smaller learning rate (a standard convention), DPO achieved significantly higher accuracy on both GSM8K (93.5%) and AIME26 (30.0%) compared to other methods. This suggests that DPO's unique weight-space geometry contributes to its superior performance in distilling reasoning, highlighting that loss function and optimizer choices jointly impact the update dynamics and final model capabilities.