New Framework Enhances LLM Reasoning by Optimizing Token Distributions

Reinforcement Learning with Verifiable Rewards (RLVR) has significantly advanced Large Language Model (LLM) reasoning, but it faces a core instability: uniform token updates can lead to "entropy collapse," causing premature convergence to suboptimal strategies, while excessive entropy maximization can result in "entropy explosion," leading to incoherent reasoning. This research proposes a solution to this dilemma. The Independent Combinatorial Tokens (ICT) framework shifts the optimization focus from simple scalar uncertainty measures to the distributional properties of token logits. By employing Jensen-Shannon (JS) divergence, ICT identifies tokens with distinct distributional patterns, which are then used as critical branching points to guide more effective exploration during LLM reasoning. Theoretical analysis confirms that selectively updating these specific tokens regulates policy concentration, reducing overall distribution uncertainty while controlling probability concentration. This dual effect prevents both over-concentrated token generation that stifles exploration and stabilizes the training landscape. Empirical results on Qwen2.5 models demonstrate significant improvements in reasoning benchmarks, with an average pass@4 gain of 4.58% and a maximum of 14.9% over existing baselines.