New Metric Evaluates LLM Agent Planning and Execution.

A new evaluation metric, named Otap (Optimal Transport for Agentic Planning), has been developed to provide a more sophisticated assessment of large language model (LLM) agents. Traditional evaluation methods often rely on simple binary success flags or exact matching against reference trajectories, which fail to capture the nuances of an agent's planning and execution process. These methods cannot differentiate between a lucky success and a well-reasoned solution, nor can they explain why a failure occurred. Otap reframes trajectory evaluation as a distance calculation between the agent's generated execution graph and a set of valid solution graphs. It employs an unbalanced fused Gromov-Wasserstein transport problem over attributed dependency graphs to achieve this. This approach yields a pseudo-metric that is inherently invariant to valid reorderings of steps and robust to redundant actions, allowing for flexibility in plan granularity and handling missing or hallucinated steps without forced matches. Experiments on controlled perturbations and public benchmarks demonstrate Otap's ability to distinguish valid from invalid trajectories, even in scenarios where semantic-only metrics perform poorly. Its accuracy is highest when dependency graphs are precisely recovered, offering a significant advancement in the rigorous evaluation of complex LLM agent behaviors.