CSPO Enhances Safe Reinforcement Learning with Constraint-Sensitive Policy Optimization.

Safe Reinforcement Learning (Safe RL) aims to maximize expected returns while adhering to safety constraints, typically modeled as Constrained Markov Decision Processes (CMDPs). While primal-dual methods are scalable for deep RL, they often suffer from delayed constraint correction, leading to undesirable oscillatory behavior and extended periods of safety violations. This paper introduces Constraint-Sensitive Policy Optimization (CSPO, a novel first-order primal-dual method designed to address these limitations. CSPO integrates local constraint sensitivity directly into policy updates. It augments the primal objective with a constraint-sensitive correction derived from the shortest signed distance to the safety boundary. This innovative approach enables smarter recovery steps back to safety, effectively compensating for the delays in Lagrange multiplier updates. As a result, CSPO significantly reduces oscillations near the safety boundary and preserves the Karush-Kuhn-Tucker (KKT) solutions of the original constrained problem. Experiments on various navigation and locomotion benchmarks demonstrate that CSPO achieves faster safety recovery and better reward preservation, leading to higher constrained returns compared to existing state-of-the-art primal-dual and penalty-based methods.