ProDVI Uses LLMs to Boost RL Sample Efficiency.

Researchers have introduced ProDVI (Programmatic Dynamics Priors for Value Network Initialization), a novel framework designed to enhance the sample efficiency of Deep Reinforcement Learning (RL). A major hurdle in RL is the need for extensive online interaction to acquire task-relevant knowledge, as agents typically start from scratch. ProDVI addresses this by utilizing the commonsense and domain knowledge embedded in large language models (LLMs). The framework prompts a code-generating LLM to produce executable Python functions that represent coarse hypotheses about environment dynamics. These functions are then used to generate synthetic transitions, which in turn serve to pretrain the state-action encoder of the value network within an actor-critic RL setup. This pretraining provides valuable dynamics-aware inductive biases before any online learning begins. Crucially, the generated programs do not need to perfectly simulate the target environment; their induced initialization can be refined through subsequent online learning from real transitions and rewards. Experiments on OpenAI Gym and DeepMind Control Suite tasks demonstrate that ProDVI effectively improves the sample efficiency of model-free RL algorithms.