DualEval Unifies LLM Evaluation with Joint Model-Item Calibration

Evaluating Large Language Models (LLMs) currently relies on two distinct data sources: static benchmarks with objective correctness labels and arena-style preference data reflecting real-world user interactions. This new research introduces DualEval, a novel framework designed to bridge this gap by performing joint model-item calibration. DualEval operates by representing both LLMs and individual evaluation items within a shared latent space. This allows the framework to simultaneously estimate an LLM's overall ability, the inherent difficulty of each evaluation item, and the "sharpness" or discriminative power of those items. The framework was applied across four diverse domains—coding, math, general domain knowledge, and everyday user queries—using 18 frontier LLMs, static benchmark labels, and reward-model scores validated against human preferences. Empirically, DualEval consistently generates reliable and balanced model rankings. Furthermore, the learned item-level profiles offer significant utility for downstream applications. These include benchmark compression, which enables more sample-efficient evaluations, and anomaly detection, useful for identifying data contamination or outlier items. By unifying static and arena-style evaluation through this joint calibration, DualEval promises more efficient, interpretable, and auditable LLM evaluation pipelines.