New ESQ-Bench Benchmark Evaluates Enterprise NL2SQL Model Performance

Current NL2SQL benchmarks often rely on simplified academic schemas and open-source SQL dialects, which do not accurately reflect the complexities of real-world enterprise databases. To address this, a new benchmark called ESQ-Bench has been developed, focusing on Oracle-first NL2SQL evaluation with systematic complexity tiers and a unique silent-divergence assessment. ESQ-Bench includes six populated schemas across various database systems, comprising 465 tables and over 164,000 rows, along with 550 gold-validated question-query pairs categorized into three complexity tiers. Initial testing with GPT-4o and Claude Sonnet 4.6 revealed significant degradation in execution match accuracy as schema complexity increased, with Claude Sonnet generally outperforming GPT-4o. A critical finding is the high rate of "silent divergence," where queries execute but produce incorrect results, reaching 73-99% among seemingly successful queries. This highlights a major challenge for deploying NL2SQL in production, as models often generate semantically wrong outputs without explicit errors, underscoring the need for more robust evaluation methods.