PPO-HSC Boosts LLM Exploration, Combats Mode Collapse.

Fine-tuning Large Language Models (LLMs) with Reinforcement Learning from Verifiable Rewards (RLVR) often leads to a problem known as "mode collapse." While RLVR effectively reinforces high-reward trajectories, it can cause models to over-optimize known solutions, stifling exploration and the ability to discover a broader range of valid reasoning patterns. To counter this, PPO-HSC (Proximal Policy Optimization with High-order Sampling Coverage) introduces a novel exploratory RL framework. It integrates a High-order Sampling Coverage (HSC) reward, which specifically encourages the model to find "low-similarity yet high-validity" reasoning patterns. This is achieved by maintaining a dynamic library of unique, verified solutions, providing a differentiable signal that rewards semantic novelty while ensuring structural rationality through a plausibility constraint. Empirical evaluations on mathematical reasoning tasks (GSM8K, SVAMP) and code generation demonstrate that PPO-HSC significantly enhances solution diversity and state-space coverage. Crucially, it achieves these improvements while maintaining or even surpassing the accuracy and syntax integrity of existing state-of-the-art RL baselines.