EMAgnet Improves Policy Gradient Self-Play in Large Games

Policy gradient methods, particularly when used in self-play, have shown great promise in solving complex two-player zero-sum imperfect-information games, sometimes outperforming specialized game-theoretic algorithms. A common regularization strategy involves targeting a uniform distribution, but this approach treats all actions equally, regardless of their strategic viability. The new technique, EMAgnet, addresses this limitation by introducing an adaptive regularization target. Instead of a static uniform distribution, EMAgnet regularizes the policy towards an exponential moving average (EMA) of the last-iterate policy's parameters. This allows the regularization target to evolve dynamically with the agent's improving strategy. Evaluations on standard and modified two-player zero-sum benchmarks, including those with significant exploration challenges and many strictly dominated strategies, demonstrated EMAgnet's effectiveness. It consistently achieved lower exploitability in most tested environments, showing clear performance gains, especially in games where strictly dominated strategies are prevalent, compared to PPO self-play with uniform-magnet regularization.