Prompt-Driven Exploration Boosts RL with LLMs.

In reinforcement learning (RL), effective exploration is crucial for policy improvement, but standard methods that inject action-space stochasticity often only yield minor variations. Escaping suboptimal policies frequently requires more significant, global perturbations that simple action noise cannot provide. Large Language Models (LLMs) and Vision-Language-Action (VLA) models offer a novel pathway by conditioning policies on natural language prompts. Modifying these prompts can induce global changes in behavior. The challenge lies in identifying prompts that lead to useful global changes, especially when rewards are sparse. Prompt-Driven Exploration (PDE) addresses this by having a Vision-Language Model (VLM) analyze rollout videos, diagnose policy responses, and then rewrite prompts to elicit better behavior. This process effectively implements posterior sampling at the prompt level, allowing the VLM to maintain and update an implicit distribution over useful prompts. PDE has been shown to enable successful policy learning from zero-reward starts and generally improves sample efficiency across various manipulation and reasoning tasks.