New Method Quantifies Uncertainty in LLM Benchmark Rankings

When evaluating large language models (LLMs), they are typically ranked on multi-task leaderboards to gauge their performance across a variety of tasks. Recently, rank confidence intervals have been proposed as a way to quantify the inherent uncertainty in these rankings by aggregating the results of pairwise hypothesis tests. This research delves deeper into the sources of this uncertainty, specifically analyzing the MMLU (Massive Multitask Language Understanding) benchmark. The study shows how hypothesis tests can be modified to account for these identified sources of uncertainty. A key finding is that the variability in LLM rankings across different subjects within the MMLU benchmark is considerable. This substantial variability underscores the importance of considering such uncertainty when comparing different LLMs or attempting to identify the truly top-performing models. The implication is that a simple point ranking might be misleading, and a more nuanced understanding of performance, including confidence intervals, is necessary for robust evaluation.