MDLMs Excel as Steerable Text World Models for Agentic RL.

The field of reinforcement learning (RL) increasingly needs diverse and specialized training environments, but hand-curated ones are limited. World models, which simulate environment states, offer a promising solution for scaling diversity. However, traditional autoregressive (AR) world models suffer from a left-to-right bias, hindering their ability to condition on globally interdependent elements like tool schemas or expected outcomes. This research formalizes text-based world modeling as a steerable transition-dynamics problem and curates a large dataset of grounded state-action trajectories. It then compares AR LMs with Masked Diffusion Language Models (MDLMs). MDLMs, through their bidirectional anchor-aware denoising, demonstrate superior coherence, groundedness, and empirically validated rollout diversity compared to AR LMs four times their parameter size, all while maintaining comparable inference latency. The paper introduces a plug-and-play GRPO training framework with deterministic state checks and shows that MDLMs achieve significant zero-shot transfer gains (up to 47% absolute) on out-of-distribution environments without environment-specific fine-tuning. Behavioral analysis and human evaluation further confirm their realism, correctness, and utility for training.