FinRCA-Bench Evaluates Financial AI Evidence Retrieval and Reasoning.

Large language models are increasingly being adopted in financial operations, but their performance in tasks like financial reconciliation is heavily dependent on their ability to access the correct evidence. This evidence is often scattered across various transactional documents and linked by relationships rather than textual similarity, making retrieval a complex challenge. A new benchmark, FinRCA-Bench, has been introduced to specifically address this issue. FinRCA-Bench is a deterministic synthetic dataset comprising 2,250 accounts-payable-to-bank reconciliation cases, including injected failures and legitimate scenarios across 14 operational tables. Crucially, root-cause labels and record-level evidence contracts are hidden from the models, allowing for an independent evaluation of retrieval capabilities versus reasoning quality. The benchmark compares various retrieval methods, from traditional Rules/SQL and classical ML to dense semantic retrieval and a novel Typed Provenance Graph Retrieval (TPGR). The study reveals that retrieval architecture profoundly influences the overall AI system performance. For instance, changing only the retrieval method can boost macro required-record recall from 0.83% to 77.70% and exact 16-class accuracy from 2.05% to 72.44%. Structural retrieval failures were found to outnumber reasoning failures by a significant margin, indicating that getting the right data to the model is often the primary bottleneck. This underscores that a correct root-cause label alone is an insufficient proxy for an auditable diagnosis, emphasizing the need for robust evidence retrieval.