New Training Paradigm Improves LLM Agent Planning with Internal World Models

Large language model (LLM) agents, despite their advanced capabilities, often struggle with long-horizon tasks because they primarily react to immediate inputs rather than proactively planning for future outcomes. Unlike humans who engage in "what-if" reasoning, current agents lack an internal world model to simulate and evaluate potential plans before committing to an action. This research aims to address this limitation by enabling LLM agents to internalize future-aware planning. The proposed solution involves training a single autoregressive model to generate both a prospective sequence of future states and an estimate of success conditioned on a given plan, essentially a textual equivalent of a Q-value. The authors identified that simply fine-tuning agents on look-ahead traces often leads to superficial imitation of foresight without true predictive understanding. To overcome this "format-capability gap," they developed a three-stage training paradigm. This paradigm includes: (i) World Model Agentic Mid-Training (WM-AMT) to embed latent predictive abilities; (ii) Format-Eliciting SFT (FE-SFT) to structure these capabilities into a usable format; and (iii) Foresight-Conditioned Reinforcement Learning (FC-RL) to refine the accuracy and utility of the generated simulations. Evaluations on search and mathematical reasoning tasks demonstrated that this approach consistently outperforms other training baselines, indicating that a capability-first pipeline is essential for achieving grounded and calibrated foresight in LLM agents.