New Metric Quantifies Cross-Task Behavioral Consistency in LLM Agents

Current evaluations of language model agents primarily focus on outcome metrics like success rate, which indicate whether an agent succeeds but not how it achieves that success or how consistently it behaves across different scenarios. This research highlights that behavioral consistency across tasks is a crucial, yet overlooked, property. To address this, the paper proposes the Behavioral Consistency Metric (BCM). BCM works by training a model to predict task success from behavioral features extracted from agent execution traces. It then derives a feature-attribution vector for each trajectory and measures the mean pairwise similarity of these vectors within an agent system. Through experiments with various LLM agents on software engineering tasks, BCM revealed that cross-task and within-task consistency are distinct and can diverge. Some agents might be reproducible on a single task but lack a stable strategy across different tasks, a distinction missed by prior work. BCM serves as a process-level reliability signal, complementing traditional outcome metrics.