PPO-Clip Algorithm Reinterpreted as Per-Sample KL Penalty

A new study provides a novel perspective on Proximal Policy Optimization (PPO), a cornerstone algorithm in reinforcement learning. Traditionally, PPO is understood through two distinct formulations: one using a clipped surrogate objective and another employing a Kullback-Leibler (KL) divergence penalty. This research reveals that the gradient of the clipped surrogate can be precisely reproduced by a KL surrogate where the penalty coefficient adapts for each individual sample, based on the importance ratio and advantage. This finding effectively unifies the two PPO variants, showing them to be fundamentally the same algorithm under a different mathematical lens. The reformulation highlights that PPO-Clip implicitly applies a per-sample penalty that acts as a step function at the trust region boundary, offering new avenues for generalizing and improving the algorithm's design. Empirical tests on MuJoCo benchmarks confirm that both forms yield indistinguishable training curves.