EnterpriseRAG Benchmarks LLM Robustness in Real-World RAG

Enterprise RAG (Retrieval-Augmented Generation) systems in production environments face substantial reliability challenges. While large language models (LLMs) might satisfy individual constraints in a RAG query, a significant "orchestration gap" exists, with only a small fraction of responses meeting all requirements simultaneously. Current benchmarks often assume ideal retrieval conditions and simple queries, failing to reflect the complexities of real-world enterprise deployments where noisy documents and multi-dimensional constraints are common. To address this, researchers have introduced EnterpriseRAG, a new benchmark comprising nearly a thousand expert-validated samples across six domains. This benchmark systematically simulates critical failure modes often overlooked in prior work, including retrieval noise, knowledge gaps, and factual conflicts, all paired with complex instructions. Evaluations of 13 state-of-the-art LLMs using EnterpriseRAG revealed a severe collapse in instruction adherence. High per-constraint satisfaction often masked a much lower holistic compliance, particularly under conditions of knowledge gaps and factual conflicts, even with advanced reasoning techniques. These findings underscore the necessity for explicit context-aware protocols and calibrated judgment in production RAG systems.