Study Compares Action Factorization Methods for RL.

Many real-world control problems, such as autonomous driving or robotics, involve complex hybrid discrete-continuous action spaces. Despite support for these spaces in various reinforcement learning (RL) frameworks, benchmark environments often default to simpler, uniform action space configurations, limiting comprehensive evaluation of factorization methods. This research addresses this gap by conducting a broad cross-sectional study. The study systematically compares different action factorization methods, including independent networks, shared encoders, VDN, QPLEX, Joint, and Auto-Regressive, across three families of RL algorithms (PPO, SAC, DQN) and three action space types (discretized, hybrid, continuous). To facilitate this, two new C++ parallel Gymnasium and PettingZoo-compliant environments, CoopPush and Hybrid-Shoot, were introduced to isolate specific challenges like state-dependent inter-action dependence. Analyzing 220 configurations, the findings suggest that branching dueling architectures offer the best balance of computational cost and performance. Auto-Regressive actions achieved the highest overall performance, while native continuous SAC performed well but at a higher computational expense. The paper also introduces VDN-PPO and PPO-MIX, which utilize a branching critic to assign credit in multi-headed PPO, demonstrating superior performance over other tested PPO factorizations.