STAIF Improves LLM Instruction Following with Stage-wise Optimization.

Large language models (LLMs) frequently struggle with adhering to intricate instructions that involve multiple explicit constraints, especially when these constraints are out-of-distribution or numerous. Current alignment methods, such as DPO, often optimize for a holistic reward signal, which can inadvertently diminish the strict satisfaction of individual constraints. To address this, researchers have introduced STAIF, a novel stage-wise optimization framework. STAIF's core innovation lies in its ability to separate the alignment process for subjective (soft) constraints from the optimization of objectively verifiable (hard) constraints. In its first stage, the framework employs preference optimization, utilizing multiple negative samples to sharpen the model's sensitivity to soft constraints. The second stage then applies Reinforcement Learning with Verifiable Rewards (RLVR) to rigorously enforce compliance with hard constraints. To facilitate this method, a high-quality bilingual dataset named STAINSTRUCT was created, comprising approximately 31,000 complex multi-constraint instructions in both English and Chinese. Extensive evaluations confirm the effectiveness of STAIF's design, demonstrating state-of-the-art performance on relevant benchmarks when compared against strong baseline models, along with robust generalization capabilities.