Multi-LLM Systems Improve Reliability with Uncertainty-Aware Trust Estimation.

Current methods for combining predictions from multiple Large Language Models often assume all models are equally reliable, which can be problematic when dealing with diverse or unreliable LLMs. This new approach addresses this by formulating multi-LLM aggregation as an uncertainty-aware trust estimation problem. It borrows from decision theory's structured expert judgment, using context-aware calibration questions to assess an expert LLM's reliability based on its probabilistic prediction quality. The method employs Cooke-style log weighting, which specifically penalizes overconfident incorrect predictions and rewards experts that are well-calibrated. Evaluations across various LLM panels, including homogeneous, heterogeneous, and contaminated setups, demonstrate that this Cooke weighting significantly improves accuracy and reliability, especially when dealing with diverse or unreliable LLMs. This suggests that effective multi-LLM aggregation requires not just combining outputs, but carefully calibrating trust based on each model's uncertainty.