LLMs Enable Prompt-Driven Exploration for Reinforcement Learning

In reinforcement learning (RL), effective exploration is crucial for a policy to improve beyond its initial behaviors. Traditional methods often rely on injecting stochasticity into the action space, which typically only yields minor variations around the current policy. Escaping a weak policy, however, often requires more global perturbations that action noise cannot provide. This research proposes Prompt-Driven Exploration (PDE), a novel RL exploration framework that leverages large language models (LLMs) and vision-language-action (VLA) models. Instead of direct action noise, the policy is conditioned on a natural language prompt. Modifying this prompt can induce global changes in the policy's rollout behavior. The core challenge then becomes finding useful prompts, especially when rewards are sparse. PDE addresses this by having a vision-language model (VLM) analyze rollout videos, diagnose the policy's response, and then rewrite the prompt to elicit improved behavior in subsequent attempts. This process effectively implements posterior sampling at the prompt level, where the VLM maintains an implicit distribution over useful prompts and updates it based on observed outcomes. Across various manipulation and reasoning tasks, PDE has demonstrated its ability to enable RL to learn successful policies even from scenarios with zero initial rewards, significantly enhancing sample efficiency.