New Meta-Benchmark Evaluates LLMs for Financial Services.

Public LLM leaderboards often prioritize general performance metrics, which may not accurately reflect a model's suitability for the specific and nuanced cognitive demands of financial services. A model excelling in general knowledge might underperform in compliance reasoning or multi-turn customer interactions within a banking context. To address this, a new meta-benchmarking framework has been developed. This framework systematically organizes 452 publicly reported benchmarks, mapping them to 41 O*NET Generalized Work Activities and further aggregating these into 38 BIAN banking business domains, covering areas like sales, operations, risk, and support. It employs a multiplicative weighting scheme that rewards benchmarks for their discriminative power, coverage, and recency, automatically de-emphasizing saturated or legacy tests. A pairwise Elo tournament, scaled by these weights, generates cross-benchmark-comparable work-activity scores and weighted business-domain scores without needing raw score normalization. The framework was demonstrated on a snapshot of 288 models from 25 organizations, providing a reproducible methodology for institutions facing similar LLM selection and governance challenges in the financial sector.