QWM: Q-Learning Enhanced with World Models for Robotics

A novel framework called QWM (Q-Learning With World Models) has been introduced to enhance the performance and sample efficiency of off-policy reinforcement learning (RL). While world models have shown promise in improving sample efficiency by predicting state changes, their successful application has largely been confined to supervised policy learning, with challenges in scaling to complex, high-dimensional problems like real-world robotics. Previous model-based RL methods often optimize policies directly on imagined rollouts, which can lead to accumulating bias and difficulties in scaling, especially with longer task horizons and visual complexity. QWM addresses this by leveraging world models specifically for test-time search over imagined trajectories, allowing it to select high-value actions during both online rollouts and evaluation. Crucially, QWM ensures that the policy and value function are trained exclusively on real transitions. This design choice effectively mitigates the compounding model bias often seen in other model-based RL approaches, while still capitalizing on the sample-efficiency benefits of predictive search. The framework demonstrated significant improvements over state-of-the-art methods on demanding manipulation benchmarks like Robomimic and LIBERO.