QPILOTS Enhances Flow Policies with Test-Time Q-Steering

This paper introduces QPILOTS, a novel approach designed to enhance the performance of flow-matching and diffusion policies in reinforcement learning (RL). These policies are powerful for generating actions, but their optimization using temporal-difference RL has been challenging due to numerical instability when backpropagating critic gradients through multi-step denoising. Existing solutions often compromise by discarding gradient information or requiring repeated policy fine-tuning. QPILOTS circumvents these issues by leaving the original policy untouched and instead steering the denoising process during inference. At each denoising step, it projects the noisy intermediate action to an estimated final clean action, where critic predictions are more reliable, and then computes the critic gradient. This technique, with variants like QPILOTS-U and QPILOTS-M, demonstrates significant improvements, achieving an average success rate of 90% across 50 tasks in a standard offline-to-online RL benchmark. Furthermore, QPILOTS successfully steers large, frozen Vision-Language Action (VLA) foundation models, matching or surpassing other inference-time methods in various manipulation tasks.