Grounded Language Planning Reduces LLM Agent Hallucinations

Large Language Model (LLM) agents, while flexible in their reasoning, often suffer from "hallucinations," where they imagine state changes that are not actually possible in the environment. This makes their planning unreliable. Conversely, traditional parameterized world models are more grounded in reality but less capable as standalone planners. This research proposes Grounded Iterative Language Planning (GILP) to combine the strengths of both. GILP integrates a compact, trained parameterized world model with the flexible reasoning of an API-based LLM agent. The parameterized model provides valid actions, predicted state changes, and risk assessments, while the LLM drafts actions and imagined state deltas. A critical "consistency gate" then compares the LLM's imagined deltas with the parameterized model's predictions. If there's a disagreement, the LLM is prompted to revise its plan. Evaluations using GPT-4o-mini demonstrated a substantial reduction in hallucinated state rates, from 17.6% to 3.5%. In simulated environments, GILP boosted success rates from 66.8% to 83.8%, with only a modest increase in LLM API calls. This hybrid approach offers a promising way to make LLM agents more reliable and less prone to generating unrealistic plans.