Safe-CRL Scales Goal-Conditioned Policy Learning with Failure Signals.

Contrastive reinforcement learning (CRL) has proven effective for goal-conditioned tasks, but its application in environments with failure termination presents a significant challenge. Existing CRL methods tend to overestimate goal-reaching values by not fully accounting for the probability mass removed by failures, leading to a bias that can reinforce unsafe actions. This "catastrophic failure bootstrapping" can result in policies that fail to learn or exhibit unsustainable behaviors. To counter this, the researchers introduce Safe Contrastive Reinforcement Learning (Safe-CRL). This framework incorporates two key corrections: "mass-weighted InfoNCE" to adjust for the overemphasis on short, surviving trajectories in critic learning, and a "log-survival-mass score" to restore the missing survival probability in policy optimization. Safe-CRL requires only a simple one-bit signal indicating failure, making it highly scalable. Experiments across various robot navigation and locomotion tasks demonstrate that Safe-CRL consistently improves survival rates and goal-reaching performance, even enabling complex failure-avoidance behaviors.