New Regret Bounds and Certificates for Average-Reward Reinforcement Learning

Comparing theoretical guarantees in average-reward reinforcement learning (RL) is challenging due to varying assumptions and numerical content across published works. This research aims to standardize this comparison by introducing a constant-aware protocol and deriving explicit, finite lower certificates for communicating Markov Decision Processes (MDPs).The authors construct a binary tree of two-state blocks to prove an exact trajectory-level Bernoulli KL divergence, keeping critical parameters like action budget and diameter explicit. This construction leads to a common closed-form envelope that improves the best published coefficient by up to 94% in certain conditions, moving from 0.015 to 0.0291.For upper bounds, the paper provides an auditable composition rule for a span-constrained optimistic learner, though it refrains from claiming a specific coefficient until further research on adaptive directional-variance and planning certificates is complete. The work also formalizes expectation conversion and constant comparability, offering a clearer framework for evaluating RL algorithms.