FinProBench Evaluates Financial AI Agents with Professional Rubrics

Evaluating AI agents in specialized fields like finance requires assessment criteria that truly reflect professional standards, which often go beyond what can be inferred from task prompts alone. Tacit knowledge and industry-specific expectations, typically embedded in practitioner deliverables, are crucial for accurate evaluation. FinProBench is a new benchmark designed for professional financial tasks, coupled with a novel methodology called Role-Grounded Rubric Construction (RGRC). RGRC systematically derives evaluation rubrics by analyzing deliverables produced by human professionals in specific roles. This four-stage pipeline—Deliverable Collection, Competency Extraction, Rubric Synthesis, and Validation—ensures that the rubrics capture nuanced quality levels and transfer across tasks within a given role. The study found that while prompt-only rubrics perform comparably to RGRC for conventional roles (where standards are well-represented in model priors), RGRC significantly outperforms them for "role-specialized" roles (99.1% vs. 78.0%). This highlights the necessity of professional grounding for evaluating AI in domains with unique, less explicit standards. FinProBench includes 1,723 curated deliverables across 57 occupations and provides an initial evaluation set of 20 tasks, demonstrating that human deliverables still rank highest, though AI systems show complementary strengths.