LLM Crowds Show Wisdom, But Contamination Skews Evaluation

The concept of the "wisdom of crowds," where aggregated judgments often surpass individual best predictions, has been explored extensively with human groups. This research examines if a similar effect occurs when the "crowd" consists of multiple large language models (LLMs). The study gathered probability estimates from 15 LLMs on 254 binary prediction questions and evaluated various aggregation techniques. The findings indicate that learned aggregation methods, such as a multilayer perceptron or logistic regression, consistently outperformed both individual LLMs and classical aggregation approaches. Notably, the logistic regression performed as well as the neural network, suggesting that the benefit comes from learning linear combinations of diverse model outputs rather than complex non-linear interactions. A critical discovery was the pervasive impact of training cutoff contamination, which dramatically exaggerated the performance difference between advanced cloud models and smaller local models. After accounting for contamination, the apparent capability gap significantly narrowed, emphasizing the necessity of clean evaluation for accurate assessment of LLM performance and ensemble benefits.