New Benchmark Reveals AI Financial Agent Instability

Traditional evaluation methods for tool-using AI agents primarily focus on the final decision, overlooking the consistency of the process used to reach that decision. This oversight can mask significant behavioral instability. A new benchmark, DFAH-Bench, has been introduced to address this gap. It specifically measures observable behavioral instability in financial AI agents by analyzing three channels: the sequence of tool calls, the evidence contacted, and the concentration of decisions. Crucially, this benchmark does not require access to the agent's internal reasoning text. Across a large-scale evaluation involving 8,127 replay episodes, 10 models, and 3 financial tasks, DFAH-Bench revealed that models can agree on decisions 95% of the time while following the same tool path only 77% of the time, highlighting an 18-percentage-point gap in process stability. Over half of the frontier models with high decision agreement showed meaningful divergence in their tool-use trajectories. The research identifies distinct behavioral profiles, including "pattern matchers" that achieve high agreement by collapsing outputs, "stable executors" with consistent processes, and "trajectory divergers" that reach similar conclusions via different paths.