Abstention Action Can Collapse Error-Penalized RL Training.

In reinforcement learning, error-penalized scoring rules (rewarding correct answers, penalizing wrong ones, and giving zero for abstaining) are increasingly used to combat issues like hallucination. However, this paper reveals a critical flaw: when abstention is treated as a discrete action, it can lead to a "collapse law." Under specific conditions, the reward gradient and the KL anchor's restoring force are throttled and die together, causing the model to drift towards refusing all questions. This makes it appear as if the model is improving (mean reward rises to zero) while its coverage collapses. The advantage estimator further exacerbates this failure, effectively replacing the designed penalty with a fixed penalty of one, shifting the learned threshold. The proposed solution is structural: train a mandatory confidence report using a strictly proper score alongside a correctness reward, and only apply abstention at deployment by thresholding this report. This ensures the report's gradient and anchor remain active, leading to improved coverage, accuracy, and calibration.