New Method Selects Optimal LLM Benchmark Prompts Without Model Scores

Evaluating Large Language Models (LLMs) often involves running them against extensive benchmarks, which can be computationally expensive and time-consuming. The challenge lies in selecting a smaller, representative subset of prompts that accurately reflects the model's performance and ranking across the full benchmark suite, ideally without needing to run the models first. This paper proposes an "evaluation-unsupervised" approach to benchmark coreset selection. Instead of relying on model evaluation outcomes, the method operates on a fine granularity, selecting subsets of prompts across multiple benchmarks. It leverages submodular subset selection, exploring various submodular functions, including determinantal point process (DPP) and facility location-based functions. Across a large suite of 35 diverse benchmarks and 18 frontier LLMs, the facility location (FL) function, using only inexpensive semantic prompt embeddings, proved highly effective. It preserved LLM scores better than twelve score-based and diversity-based baselines across various coreset budgets. The approach also showed strong performance even when some model scores were available, matching or outperforming state-of-the-art baselines on MMLU and MTEB leaderboards while being significantly cheaper to compute. This suggests submodularity is a robust tool for benchmark compression.