Oyster-II Enhances LLM Safety with Constructive Reinforcement Learning.

Large Language Models (LLMs) face a persistent challenge in balancing helpfulness, trustworthiness, and safety. Traditional safety alignment often relies on refusal-oriented strategies, which, while mitigating harmful content, can also prevent LLMs from constructively addressing legitimate user needs, even for sensitive queries. Building on the "constructive safety" paradigm, Oyster-II aims to provide thoughtful, response-oriented safety. This research identifies two key limitations in the predecessor, Oyster-I's Supervised Fine-Tuning (SFT) approach: poor safety generalization to out-of-distribution scenarios and "safety chain-of-thought (CoT) over-generalization," where safety reasoning is excessively applied to benign queries, degrading user experience. To overcome these, Oyster-II proposes a reinforcement learning (RL)-based framework, utilizing a Zero-RL paradigm combined with a multi-stage RL strategy. Extensive benchmarks demonstrate that Oyster-II significantly surpasses both Qwen3-14B and Oyster-I in safety dimensions, achieving cross-scale performance comparable to much larger models like Qwen3-Max and Qwen3.5-397B.