RENEW Uses Human Preferences to Improve World Models in RL

World models are crucial in offline reinforcement learning (RL) for improving sample efficiency and generating new experiences. However, they are prone to "model exploitation," where the model generates unrealistic dynamics in regions with sparse data, leading to poor agent performance. Existing solutions often involve costly expert demonstrations or overly conservative algorithms that limit generalization. This paper proposes a new approach to directly address this exploitation. The proposed method, Dynamics Learning from Human Feedback (DLHF), formalizes the use of human preferences over imagined rollouts to supervise world model dynamics. Humans can intuitively identify unrealistic dynamics, providing a powerful signal for correction. To overcome the sample inefficiency of naive DLHF, the researchers introduce RENEW. RENEW leverages epistemic uncertainty to strategically focus finetuning on areas where the model is most exploitable. Evaluations across various Jumanji and classic control environments demonstrate that RENEW significantly improves sample efficiency, mitigates catastrophic forgetting, and effectively reduces exploitation in pretrained world models. These findings suggest that human preferences can directly supervise world model dynamics, offering a promising new avenue for robust model-based RL.