SFT Overtraining Predicts Rank Inversion in RLHF Models

In the development of large language models, the standard practice of selecting the Supervised Fine-Tuning (SFT) checkpoint with the highest initial performance (e.g., pass@1) for subsequent Reinforcement Learning from Human Feedback (RLHF) can be counterproductive. This study reveals that overtraining during the SFT phase can lead to a phenomenon called "rank inversion," where models that initially perform better ultimately achieve worse results after RL. The core mechanism behind this failure is identified as an entropy collapse within the SFT rollout distribution, especially when dealing with binary rewards. When SFT compresses this distribution too much, the expected variance of advantage within groups diminishes, leading to a lack of meaningful signal for the RL algorithm. This makes it difficult for the RL phase to effectively improve the model. Experiments with Qwen2.5-Coder-3B and DeepSeek-Coder-6.7B models demonstrated this effect. For Qwen, increasing SFT depth led to higher pre-RL pass@1 but a significant drop in peak GRPO pass@10, with pre-RL entropy positively correlating with the RL outcome. A two-stage diagnostic combining pre-RL entropy triage with an early GRPO entropy monitor is proposed to identify and stop failing runs early, suggesting that simple regularization techniques may not fully mitigate this issue.