Game Theory Optimizes LLM Fine-Tuning for Reward-Retention

Reinforcement learning (RL) fine-tuning is a common technique for improving language model performance while controlling divergence from an original policy, typically using a KL-regularized objective. However, setting the regularization coefficient for this objective often involves heuristic choices or costly hyperparameter searches. This research introduces a game-theoretic framework that provides a principled, statistical interpretation for this trade-off. It models a sequential game where an agent maximizes reward while a monitor tests for policy deviations. The resulting equilibrium policy corresponds to a KL-regularized RL problem with an optimal regularization parameter, which effectively maximizes reward per unit of statistical distinguishability. This method, drawing on concave-convex fractional programming, offers a flexible way to integrate optimal coefficient learning into standard fine-tuning pipelines, demonstrating competitive reward-retention trade-offs in continual learning settings.