Entropy-Regularized RL for Stackelberg Games in Dynamic Models

Solving Stackelberg differential games (SDGs), which model hierarchical decision-making in continuous-time stochastic environments, is computationally intensive, especially in high-dimensional systems. Traditional methods like dynamic programming and Hamilton-Jacobi-Bellman-Isaacs (HJBI) equations face significant challenges. This research introduces an entropy-regularized reinforcement learning (ERRL) approach specifically for linear-quadratic SDGs (LQ-SDGs) within a continuous-time diffusion framework that incorporates Markovian regime switching. A key innovation is the derivation of exploratory, weakly-coupled HJBI equations, which, through entropy regularization, encourage stochastic policies that actively avoid suboptimal outcomes. Neural networks are integrated to approximate regime-dependent value functions and efficiently solve the high-dimensional partial differential equations. A novel sampling technique further enhances computational tractability. Numerical results confirm the framework's effectiveness in escaping suboptimal traps via exploratory policies, highlighting the crucial role of entropy regularization and neural network approximations for robust hierarchical decision-making under abrupt environmental shifts.