New Actor-Critic Algorithm Accelerates Unregularized Reinforcement Learning Convergence.

This paper presents a novel approach to Natural Actor-Critic (NAC) algorithms, a cornerstone in reinforcement learning. Traditionally, entropy regularization is used to stabilize and speed up these methods, but its impact on the convergence of the unregularized objective has been less understood, often requiring complex double-loop architectures. The new algorithm simplifies this by using a single-loop, entropy-regularized NAC. A key innovation is the use of an uncentered critic, which maintains stability even as the training policy becomes deterministic and the Fisher information matrix might otherwise degrade. The research explores both stochastic and deterministic optimization regimes, demonstrating accelerated unregularized convergence rates by introducing an "Exponential Translation" mechanism. This mechanism effectively maps the regularized performance gap to the unregularized one, leading to significant improvements in convergence speed, particularly in tabular settings where it surpasses previous statistical barriers.