Diverse LLM Crowds Outperform Larger, Redundant Ones

This paper introduces a novel framework for leveraging the "wisdom of crowds" principle with large language models (LLMs) for future prediction tasks. The core insight is that simply increasing the number of LLMs in a crowd does not guarantee better performance if the models exhibit redundant behaviors. Instead, the research emphasizes the importance of behavioral diversity. The proposed framework characterizes LLMs based on their reasoning traces on independent development tasks. These models are then clustered by their behavioral similarity, and a small set of representative models is selected to form a diverse crowd for collective prediction. This approach aims to maximize the unique perspectives brought by each model. Evaluations involving 25 LLMs across multiple benchmarks showed that a small, three-model medoid crowd, selected based on K-means++ behavioral clustering, significantly outperformed a conventional voting approach using all 25 models. This method not only improved prediction accuracy but also drastically reduced model calls by 88% and inference costs by approximately 80%, highlighting the efficiency and effectiveness of representative behavioral diversity.