AI Uncertainty Fusion Improves Trust, Not Prediction, in Legal Cases

A study investigated the effectiveness of integrating various uncertainty tools, such as evidence graphs, Bayesian odds updating, and conformal prediction, into large language model (LLM) pipelines for predicting legal case outcomes. Using 1,000 real European Court of Human Rights cases, researchers compared raw LLM performance against LLMs routed through these fusion pipelines and a term-frequency baseline. The primary finding was that while these pipelines did not improve prediction discrimination (AUROC), they significantly enhanced "calibrated trust." The research revealed that naively combining LLMs with Bayesian-odds and Dempster-Shafer fusion more than doubled calibration error, suggesting a prior-mismatch issue. Notably, Dempster-Shafer fusion was found to be actively unsafe on long chains, confidently committing to incorrect labels, leading to a recommendation for its removal. The true value of these pipelines emerged in their operational utility: when tuned and routed through a conformal selective-prediction layer, the system could effectively decide which cases to automate and which to escalate for human review, achieving high accuracy with low error rates for automated cases. This indicates that for legal AI, the benefit of such pipelines lies in providing calibrated trust and enabling intelligent workflow automation, rather than merely sharper predictions.