New RL Method Improves Embodied World Models with Robust Rewards

Reinforcement Learning (RL) has shown promise in refining world models, but current methods often rely on conservative exploration strategies, limiting the discovery of diverse behaviors and richer dynamics. A core challenge is the lack of reliable verification mechanisms, which makes broader exploration susceptible to "reward hacking"—where policies exploit imperfect reward functions without achieving genuine task improvement. This paper tackles these limitations by proposing a two-pronged approach for embodied world models, where physical plausibility and task completion provide a rigorous testing ground. On the verification side, it introduces "Reward as an Agent," an agentic reward framework that actively evaluates generated behaviors to provide robust reward signals and mitigate reward hacking, even under distribution shifts. For exploration, the research presents "Dynamic-Aware Rollout Diversification" through DynDiff-GRPO. This method explicitly expands action-space exploration to diversify trajectories, broaden state-action coverage, and encourage richer embodied behaviors beyond typical conservative rollout regimes. By combining these two innovations, the approach enables more reliable RL with significantly diversified sampling, leading to substantial accuracy gains across multiple open-source world models.