Q-CARE Improves RAG Evaluation with Query Coverage and Claim Verifiability

Evaluating Retrieval-Augmented Generation (RAG) systems effectively across a wide range of user queries, from simple fact-seeking to complex explanatory requests, has been a significant challenge. Current evaluation frameworks often struggle to provide consistent and detailed diagnostics. This new research introduces Q-CARE, a novel framework designed to offer query-agnostic and fully reference-free assessment of RAG performance. Q-CARE achieves fine-grained evaluation by breaking down user queries into smaller sub-queries and decomposing generated answers into atomic claims. It establishes a unified evaluation principle based on two core concepts: query coverage, which assesses how well the retrieved information addresses all aspects of the query, and claim verifiability, which checks the factual accuracy of each atomic claim against the retrieved evidence. This approach yields new retriever metrics like C-Prec@k and C-nDCG@k, and generator metrics such as Completeness, Conciseness, and Verifiableness. Evaluations on a human-annotated benchmark, spanning eight diverse datasets, demonstrate that Q-CARE achieves a higher correlation with human judgments compared to four existing RAG evaluation metrics, including RAGEval and RAGChecker. This indicates its effectiveness as a reliable and automated framework for diagnosing RAG system strengths and weaknesses.