Decentralized Wagering Aggregates LLM Predictions Effectively.

When combining predictions from multiple large language models (LLMs), especially in decentralized environments where models have private information, determining how to weight each model's input can be challenging. Existing methods often struggle with strategic reporting or require centralized control. This research introduces WALLA (Wagering Mechanisms for LLM Aggregation), a novel framework designed to address these issues. WALLA allows each LLM to report both its prediction and a learned "wager." These wagers then serve as weights for aggregating the predictions. The mechanism is designed with a unique payout function that ensures models are incentivized to report their true predictions and that their wagers accurately reflect their expected performance advantage. A key benefit of WALLA is its ability to enable decentralized learning of wager policies without requiring optimal predictions, making it robust to strategic behavior. Experiments on various benchmarks, including question-answering and forecasting, demonstrate that WALLA achieves predictive performance comparable to centralized aggregation methods while offering the advantages of decentralization, uncertainty awareness, and incentive compatibility.