Multi-Role RL Improves Symbolic Planning from Solver Feedback.

Converting natural language instructions into precise, executable symbolic specifications (like PDDL) is crucial for reliable planning, but large language models (LLMs) often struggle with this without extensive, costly PDDL annotations. Furthermore, LLMs can sometimes exploit solver success in ways that are not semantically faithful to the original instruction. This research investigates how to learn faithful natural-language-to-PDDL formalization solely from solver feedback, bypassing the need for human-written demonstrations. The authors propose a solver-grounded multi-role reinforcement learning framework where a single language model takes on three distinct roles: Actor, Judge, and Editor. The Actor is responsible for proposing PDDL specifications. The Judge provides a quality signal calibrated by the solver, evaluating the proposed plans. The Editor then performs bounded refinement based on diagnostic feedback. Evaluated on PlanBench, this method dramatically improves average planning success from 35.5% (for LLM+P) to 70.8%. Crucially, it achieves 66.3% faithful success, meaning the generated plans accurately reflect the original intent, and significantly reduces semantic drift to just 6.4%. These results demonstrate that structuring solver feedback into these generation, verification, and repair roles enables more scalable and faithful symbolic planning without the need for manual annotations.