Bayesian Accuracy Mitigates Length Bias in LLM Evaluation.

When evaluating Large Language Models (LLMs) on multiple-choice benchmarks, a common issue is "length bias." Standard scoring methods, which sum log-probabilities over tokens, inherently penalize longer candidate answers compared to shorter ones. While a common heuristic to mitigate this is to normalize scores by completion length, empirical evidence shows this often over-corrects, introducing a new bias towards longer answers. The researchers conducted an analysis of these scoring rules, characterizing when both standard and length-normalized accuracy are appropriate and how their respective length biases are influenced by the distribution of completion lengths. Motivated by these findings, they propose "Bayesian accuracy." This novel scoring rule calculates the posterior probability of each candidate answer, incorporating an explicit prior distribution over answer length. This approach effectively removes linear length effects from the evaluation. Bayesian accuracy is designed as a direct replacement for existing likelihood-based multiple-choice evaluation methods, requires no additional computational passes, and consistently demonstrates lower empirical length bias than both standard and length-normalized accuracy across various benchmarks and few-shot settings.