S2T-RLHF Improves Stability in Preference-Based RLHF.

Researchers have developed S2T-RLHF, a novel framework aimed at improving the stability of Reinforcement Learning from Human Feedback (RLHF) in large language models. A common challenge in RLHF is the instability caused by propagating a single, sequence-level scalar reward to fine-grained token-level policy updates, leading to ambiguous credit assignment. S2T-RLHF addresses this by introducing a granularity-aware principle for hierarchical credit assignment. It argues that overly fine-grained reward refinement can amplify noise when preference signals are only defined at the response level. Instead, it proposes using sentences as a natural intermediate granularity, balancing semantic coherence with robustness to token-level noise. The framework first allocates sequence-level preference rewards across sentences and then applies a bounded token-level refinement within each sentence. This process avoids the need for reward-model retraining or additional token-level supervision. Experiments across various datasets and optimization settings demonstrate that S2T-RLHF significantly enhances training stability and robustness while maintaining competitive preference alignment.