New AI Method Learns to Improve Multi-Step Plans

Humans excel at problem-solving by formulating plans and mentally simulating outcomes using an internal world model. While machine learning has produced world models capable of predicting action sequence outcomes, the process of refining candidate plans has not been fully learned. Existing planners either rely on manual design, are distilled from pre-designed optimizers, or are trained solely to inform an amortized policy rather than to directly revise the plan itself.This research introduces Reinforced Planning (RP1), a groundbreaking method that learns search strategies by reinforcing effective search rules into a neural planner. RP1 uniquely learns both to evaluate imagined outcomes via a critic and to enhance multi-step plans through an optimizer trained entirely offline using simulated world-model rollouts. This makes RP1 the first method to fully learn how to improve complex, multi-step plans.RP1 is also highly versatile, capable of being trained independently and integrated with any pre-trained latent world model. Evaluated across visual navigation, arm reaching, and robotic manipulation tasks using two different world-model backbones, RP1 demonstrated substantial performance gains over traditional hand-designed search algorithms. It achieved near-perfect success in several scenarios, utilizing significantly fewer world-model rollouts (1,000 times less) and operating up to 67 times faster than the strongest alternative under concurrent planner inference.