Boundary-Seeking Policy Gradient for Safe RL Introduced

Safe reinforcement learning (RL) aims to maximize rewards while adhering to specific safety constraints. In Constrained Markov Decision Processes, the optimal policy often lies precisely on the constraint boundary when that constraint is active. However, conventional gradient-based safe RL methods frequently converge to a policy within the feasible interior, failing to fully exploit this boundary structure. This research proposes Boundary-Seeking Policy Gradient (BSPG), a novel first-order optimization method designed to address this limitation. BSPG's update rule integrates two components: a tangential component that improves the reward function while maintaining the cost to the first order, and a signed, residual-driven normal component that actively guides the policy towards the active constraint boundary from either side. This combined direction can be expressed in an algebraic Lagrangian form without requiring a learned dual variable. Under specific conditions, BSPG ensures that the constraint residual converges to zero, meaning the policy accurately tracks the boundary. The tangential component guarantees reward ascent on the boundary, and any convergent parameter sequence satisfies the KKT conditions at the limit. This analysis complements existing work that primarily certifies feasibility but doesn't characterize the constraint value at convergence. Experiments on a standard Safety-Gymnasium navigation task demonstrate that BSPG achieves higher rewards and maintains tighter adherence to the safety boundary compared to baseline methods.