LLM Agreement Not Always Accuracy: Auditing Confidence Signals

Researchers conducted an extensive study involving 53 different LLM configurations, generating over 265,000 samples to investigate whether agreement among LLM judges or a single model's self-consistency reliably indicates correctness. The findings challenge the common assumption that consistency equates to accuracy, demonstrating that LLMs can agree due to shared biases, memorized patterns, or prior probabilities rather than factual truth. The study found that agreement is a positive but weak predictor of correctness, with its utility varying significantly by context. It proved most useful for mid-tier models and for optimizing computational resource allocation. However, for the most advanced "frontier" models, high agreement often correlated with over-confidence, where models were highly consistent but frequently incorrect. This over-confidence was observed across different model families, suggesting that self-consistency should not be treated as a standalone confidence score.