Controlling Critic Complexity Improves Actor-Critic RL Diagnostics

Actor-critic methods, a cornerstone of reinforcement learning, heavily rely on the quality of their learned critics. However, the quality of these critics is typically assessed indirectly through metrics like return or temporal-difference error. This new study proposes "critic complexity" as an additional, direct diagnostic and intervention dimension, offering a deeper insight into the training process. The researchers quantify critic complexity using spectral effective-rank entropy, which summarizes the singular-value distributions of critic weight matrices. Across experiments with TD3 and PPO algorithms, they show that critic complexity can be measured throughout training and is systematically linked to training behavior, although this relationship varies depending on the algorithm, task, and hyperparameters. Furthermore, the study demonstrates that critic complexity can be actively controlled by adding a spectral-entropy penalty to the critic loss, reliably altering the targeted spectral quantity. While the impact on overall return is task-dependent, this work establishes a new way to observe and influence the internal dynamics of actor-critic models.