New Method Enhances LLM Safety and Helpfulness at Inference Time

Aligning large language models (LLMs) to be both helpful and harmless is a significant challenge. While fine-tuning is effective, it's computationally expensive. Inference-time alignment offers a more efficient alternative by guiding a frozen model's decoding process with auxiliary reward signals. However, existing methods often struggle to incorporate explicit safety constraints without manual tuning of penalties. This paper introduces Lagrangian Reward Augmentation (LARA), a novel framework that addresses this limitation. LARA transforms a constrained optimization problem, involving both a reward model and a cost model, into a simpler one-dimensional convex problem. By estimating a dual variable on a small calibration set, LARA generates an augmented reward signal that can be seamlessly integrated into various inference-time alignment techniques, such as Best-of-N reranking or token-level reward-guided decoding. Evaluations show that LARA significantly improves the balance between helpfulness and harmlessness. When combined with Best-of-N reranking, LARA's performance approaches that of more resource-intensive fine-tuning methods, demonstrating a principled and efficient way to enforce safety constraints during LLM inference.