CANDI-QA Benchmark Evaluates LLMs for Niche Domain QA

This paper introduces CANDI-QA (Contextual Alignment for Niche Domains Question Answering), a novel benchmark dataset specifically created to assess the capabilities of large language models (LLMs) in specialized domains such as medical diagnostics and financial advisory. Traditional question-answering benchmarks often fail to capture the nuanced contextual grounding, user awareness, and deep domain understanding essential for high-stakes applications. CANDI-QA comprises expert-curated question-answer pairs categorized into "Information Assistance Questions" for direct factual extraction and "Applied Inference Questions" requiring multi-hop reasoning and situational inference. The evaluation of over ten diverse LLMs, alongside a neuro-symbolic baseline (MTSS-Net), highlighted significant limitations of current LLMs in achieving contextual alignment within niche domains without augmented contextual or symbolic integration. This benchmark serves as a critical tool for advancing research toward more robust and trustworthy AI systems for high-stakes professional environments.