SafeExplorer Reduces Falls in Robot RL Training

Training reinforcement learning agents on physical robots is inherently risky and costly, as falls can damage hardware and cannot be easily undone. Current approaches often use recovery policies to prevent damage when an agent leaves a safe region, but these interventions can introduce silent biases into on-policy updates, making learning less efficient or effective. Researchers have developed SafeExplorer, a modification for Proximal Policy Optimization (PPO), to address this challenge. Its core innovation is an unbiased policy-gradient estimator that only uses the score function during safe timesteps and avoids evaluating the recovery policy's density. This ensures validity even with deterministic recovery policies, where traditional importance sampling fails. SafeExplorer also incorporates additional components to accelerate learning, such as a closed-form value for recovery-triggering states and an imitation loss for successful recovery actions. Benchmarking across multiple environments showed SafeExplorer drastically reduced training-time falls (by factors of 26x to 233x) while achieving comparable or superior final rewards compared to standard PPO.