Study Reveals Key Actor-Critic Design Choices for Real-World RL Reliability

Reinforcement learning (RL) is increasingly being considered for controlling critical real-world systems, from industrial processes to autonomous vehicles, where reliability and limited tuning budgets are paramount. Actor-critic algorithms, a popular class of RL methods, involve numerous design decisions, such as how policies are updated, how action distributions are represented, gradient estimation techniques, and the relative update frequency of policy versus value estimators. To provide practical guidance, researchers conducted an extensive empirical study involving over 33,000 experiments. Using a control task derived from a real water treatment plant, they analyzed how these various components influence the variability across runs and the sensitivity to hyperparameters. The study revealed that commonly used default configurations, such as Gaussian action distributions combined with pathwise gradient estimators, are among the least reliable options. In contrast, using bounded action distributions and adaptive update schedules demonstrated significantly greater robustness across a wide range of settings. These findings offer crucial empirical insights for practitioners in scientific and engineering fields, helping them make more informed component-level decisions when adapting actor-critic methods for new real-world control applications.