C3RL Improves LLM Confidence Calibration for Adaptive Scaling

This research addresses a significant issue in large language models (LLMs) trained with reinforcement learning (RL): while RL improves performance, it often leads to poor calibration between an LLM's expressed confidence and its actual accuracy. This can result in overconfident hallucinations when the model is uncertain. To counter this, the authors propose Correctness and Confidence Calibration Reinforcement Learning (C3RL). C3RL is a novel RL algorithm that incorporates a multi-faceted reward system, incentivizing not only correctness but also accurate confidence expression and adherence to dataset-informed reference accuracy. Comprehensive evaluations across eight text and multimodal datasets demonstrate that C3RL significantly improves calibration without compromising the model's performance, outperforming existing state-of-the-art methods in both metrics. Leveraging the well-calibrated verbalized confidence from C3RL, the researchers also introduce Confidence-based Adaptive Test Time Scaling (CAS). This inference-time strategy dynamically allocates computational resources based on the model's confidence in its response. Experiments show that CAS surpasses majority voting in performance while dramatically reducing the inference budget by up to 12.33 times, paving the way for more reliable and resource-efficient LLM deployments.