Predicting Text-to-SQL Correctness: LLM Judges Outperform Self-Consistency

This research explores how to reliably predict the correctness of SQL queries generated by AI from natural language prompts, especially for complex multi-table scenarios. The study evaluates various signals to determine their effectiveness in distinguishing correct queries (those that produce the same result as a human-written reference) from incorrect ones. Signals tested included string, structural, and execution self-consistency, schema relevance, query executability, and white-box log-probability. The findings indicate that while black-box signals like self-consistency offer some predictive power, their AUROC scores hover around 0.61 to 0.68. The most effective predictors were verification-based methods, specifically using large language model (LLM) judges. GPT-4o-mini and Claude judges achieved AUROC scores of 0.72 and 0.78, respectively. An ensemble of judges from different providers further boosted performance to 0.82 AUROC, demonstrating well-calibrated probabilities and supporting effective selective abstention strategies. The study also examined training fine-tuned verifiers, which performed well in-distribution but struggled with unseen schemas, suggesting that cross-schema generalization relies more on model scale and reasoning capabilities than fine-tuning. In practice, robust correctness prediction for text-to-SQL applications currently depends on powerful, frozen reasoning models acting as verifiers.