Supervised UQ Ensembles Boost LLM Hallucination Detection Robustness

This research systematically examines the robustness of supervised uncertainty quantification (UQ) ensembles for detecting hallucinations in large language models (LLMs) within closed-book settings. While prior work suggested combining UQ signals via learned ensembles, a comprehensive robustness analysis was lacking. This study addresses that gap by training a classifier over diverse UQ-based scorer outputs using small, domain-specific labeled datasets, then applying it to out-of-sample hallucination classification without external tools or reference documents. The investigation spans four LLMs, nine datasets, and three generation regimes (short-form QA, long-form generation, and code generation). Key findings indicate that supervised ensembles outperform the best individual UQ scorer in nearly all tested scenarios, achieving gains with as few as 100 labeled instances. Furthermore, these ensembles maintain most of their advantage even when transferring to in-domain datasets with distribution shifts. The study also notes that sampling-based black-box ensembles are nearly as effective as full ensembles, while single-generation white-box ensembles offer limited benefits.