Warp RL Reshapes Policies for Robot Dynamics Adaptation

Researchers have introduced Warp RL, a novel approach to adapt pretrained robot policies to new dynamics, addressing key limitations of traditional residual reinforcement learning. While residual RL adds corrections to actions, it struggles when the underlying dynamics shift significantly, as it cannot alter the fundamental shape, scale, or state-dependent geometry of the base policy's action distribution. This can lead to suboptimal performance, even worse than the unadapted policy. Warp RL overcomes these issues by replacing additive corrections with an invertible, state-conditioned transformation of the base policy's action distribution, instantiated using monotonic rational-quadratic spline flows. This method strictly generalizes additive residual correction, preserves identity initialization, and provides a structured adaptation space suitable for various optimization techniques. Experiments on ManiSkill3 manipulation tasks demonstrate that Warp RL matches residual correction when simple translation suffices, but significantly outperforms it when adaptation requires reshaping the action distribution. It also showed faster task completion in a real-robot sim-to-real peg-insertion task.