World Models Accelerate and Improve Automatic Research Agents

Automating empirical research with AI agents has gained traction, but scaling the training of these "AutoResearch" agents faces a fundamental bottleneck: the high cost of environment execution. While agent generation can be batched for efficiency, each execution requires an exclusive sandbox and real machine time, making it the dominant cost as trajectories grow longer. This tension limits the scalability of reinforcement learning (RL) for these agents. To overcome this, researchers propose World Model RL (WMRL), which replaces the expensive real-world environment execution with a learned "world model." This approach effectively removes the execution bottleneck, allowing for much faster training. Recognizing that world models can be imperfect, WMRL incorporates two key mitigations: Online Debiasing to correct for systematic errors and Inverse-Variance Denoising to suppress noise in the model's rewards. Theoretically, these mitigations are proven to improve convergence guarantees. Empirically, WMRL accelerates training by 3-4x across various tasks and agent scales, while consistently outperforming standard RL baselines. Notably, WMRL-trained 4B and 9B agents surpassed much larger 48B and 120B open-weight agents on held-out benchmarks. The method's generalizability was also demonstrated by its successful application to post-training embodied Visual Language Action (VLA) policies, highlighting its broad potential.