New Benchmark Evaluates AI Agents in Mortgage Loan Origination

A new research paper introduces MortarBench, a novel benchmark specifically designed to evaluate the performance of AI agents in the critical process of mortgage loan origination. This benchmark addresses a significant gap in the industry, as firms are increasingly deploying AI to assist human loan officers without a standardized method for assessing their capabilities and risks. The study utilized a sophisticated financial data synthesis and mutation pipeline to create a diverse set of real-world scenarios and edge cases for testing. Initial findings indicate that state-of-the-art large language models (LLMs) currently perform inadequately, achieving a maximum exact match accuracy of only 77.1%. Furthermore, the research uncovered systematic biases within LLMs, particularly concerning their perception of non-English names, which could lead to unfair or inaccurate assessments. To mitigate these identified weaknesses, the researchers developed CRIT, a confidence calibration framework. Implementing CRIT significantly improved the accuracy of LLMs to 80.5% and demonstrated enhancements in risk management steering while actively reducing the observed biases. This work highlights the need for robust evaluation tools and bias mitigation strategies as AI adoption grows in sensitive financial applications.