EVOM Automates Actor-Critic Architecture Design for Reinforcement Learning

The EVOM framework introduces an innovative approach to automating the design of neural network architectures for actor-critic reinforcement learning (RL). Traditionally, these architectures are crafted manually, a process that is both time-consuming and challenging due to the vast design space and the need to train each candidate for evaluation. EVOM tackles this by framing architecture search as a bi-level optimization problem. The framework operates with an inner loop that handles the training of network weights using a low-fidelity Proximal Policy Optimization (PPO) algorithm. Crucially, an outer loop drives the meta-evolution of architectures. This outer loop is powered by a Large Language Model (LLM) acting as a design agent, which is entirely decoupled from the policy execution and environment control. This separation allows the LLM to focus purely on architectural innovation. Experimental results demonstrate that EVOM surpasses manually designed baselines, LLM-guided random search, and other state-of-the-art LLM-guided programmatic policy search methods. It achieved superior performance on challenging benchmarks like Ant-v4 and HalfCheetah-v4. Ablation studies confirmed that both the meta-evolutionary loop and the LLM Design Agent are essential components contributing to the framework's overall effectiveness.