LLM-as-a-Judge Evaluations Show Significant Instability and Bias

The widespread adoption of "LLM-as-a-Judge" for ranking model outputs, training reward models, and populating leaderboards has prompted a critical examination of its reliability and potential biases. This research systematically investigates the consistency of such evaluations by conducting repeated identical trials across 29 tasks and 10 categories, using two OpenAI judge models (GPT-4o-mini and GPT-4.1-mini). The study involved 50 pairwise and 50 pointwise trials per question, complemented by ablations for temperature and prompt sensitivity. The findings reveal substantial instability: pairwise preferences flipped on average 13.6% of the time, with nearly a third of questions exceeding a 20% flip rate, and one question showing a 56% flip rate. Furthermore, GPT-4o-mini exhibited a significant first-position bias, favoring the first option in 72% of cases. Despite small mean pointwise score gaps, judges frequently declared a winner even when their scalar scores indicated minimal quality differences, highlighting a discrepancy between pairwise and pointwise assessments. Beyond within-judge inconsistency, cross-judge agreement was only 76%, and minor changes to prompt templates altered majority outcomes in 25% of cases. While deterministic decoding reduced inconsistency, it did not eliminate it. A reliability curve analysis suggests that, on average, 11 repeated trials are needed to recover the 50-trial reference verdict with 95% probability, increasing to 15 for high-variance questions. These results underscore that single-trial LLM judging is often too noisy for critical evaluations, advocating for standard practices like multi-trial aggregation, position randomization, and explicit uncertainty reporting.