New Method Ensures Private, Regret-Optimal LLM Inference Alignment

Inference-time alignment strategies, such as Best-of-N (BoN) sampling, are commonly used to refine large language model (LLM) outputs. However, these methods face two significant challenges: the potential for reward hacking, where the model exploits flaws in the reward system, and a lack of privacy protection for the sensitive human preference data used to train the reward models.This paper proposes a unified solution by introducing calibrated noise to reward scores before selection. The first method, Private Best-of-N (PrivBoN), demonstrates that Gumbel noise can simultaneously provide differential privacy and implement KL-regularized alignment. Crucially, beyond a certain privacy budget threshold, the noise required for privacy becomes the optimal regularization, meaning privacy comes at no additional alignment cost.To address the dependency on an unknown coverage coefficient, the authors also present Private Inference-Time Pessimism (PrivITP). This approach combines chi-squared regularized rejection sampling with a two-phase Gaussian mechanism, achieving ex-post differential privacy with a cost independent of the number of responses. Experiments confirm that both PrivBoN and PrivITP are scaling-monotonic, unlike standard BoN, and PrivITP often outperforms PrivBoN, especially in strong-privacy scenarios.