Robust Peak-Cost Constrained RL Enhances Safety in AI Systems

Traditional Constrained Markov Decision Processes (CMDPs) focus on minimizing *expected cumulative* costs, which is insufficient for safety-critical applications where a single large cost violation can be catastrophic. This study introduces Robust Peak-cost Constrained Reinforcement Learning (RP-CRL), an approach designed to maximize rewards while strictly limiting the *maximum* cost incurred at any point during a trajectory. A key challenge in this domain is the potential for simulator-to-real-world discrepancies in transition dynamics. RP-CRL explicitly addresses this "sim-to-real" mismatch by incorporating a robust formulation. Unlike standard CMDPs, the research highlights that peak-cost constrained MDPs may not always have a zero duality gap, complicating traditional Lagrangian-based solutions. To overcome these issues, the authors developed a surrogate optimization framework combined with a robust value estimation method based on integral probability metrics. This framework is theoretically proven to achieve the same robust reward as the original problem while ensuring constraint violations are kept within a small epsilon bound, even under dynamics perturbations. Experiments confirm that the proposed method effectively enforces safety and maintains strong reward performance in uncertain environments.