DirEAG Calibrates LLM Confidence in Mathematical Reasoning

Relying on large language models (LLMs) for mathematical reasoning necessitates accurate confidence estimation, yet the self-reported confidence from these models is often poorly calibrated and difficult to interpret. When an LLM is prompted multiple times for the same problem with varying confidence-steering instructions, the resulting answer-confidence pairs contain valuable uncertainty information, but their scales can shift unpredictably across different prompts, models, and datasets. Existing black-box uncertainty methods typically focus on output variation, such as answer agreement or entropy, rather than modeling the numerical meaning of the self-reported confidence. To address this, DirEAG (Dirichlet Evidence Aggregation) proposes a novel approach. It transforms each elicited answer-confidence observation into calibrated soft evidence, distributed over candidate answers and an additional "null" state, which accounts for cases where none of the provided candidates are correct. Experiments conducted on mathematical reasoning benchmarks like GSM8K, SVAMP, and GSM-Hard, using models such as Qwen, Mistral, and Gemma, demonstrate that DirEAG consistently achieves superior calibration compared to direct confidence averaging or heuristic aggregation methods, all while preserving strong answer selection performance. The research also highlights that both evidence aggregation and a final binary calibration step are crucial for effectively tackling different aspects of the calibration challenge.