New Evaluation Method Improves Agentic System Assessment.

Evaluating agentic systems, especially in offline settings, often relies on simple terminal success metrics. This common practice, however, discards valuable information about an agent's partial progress and the time taken to achieve results, frequently leading to numerous tied comparisons between different systems. Such ties reduce the effective sample size and weaken the ability to discern performance differences. Researchers have introduced a novel approach called preference-based trajectory evaluation to address these limitations. Instead of just looking at final success, this method directly compares entire trajectories by considering temporal preferences related to progress and the time-to-return profiles. This provides a much richer and more nuanced assessment of an agent's performance. Experiments across various agentic and interactive benchmarks reveal that while standard success-based metrics result in approximately 75% tied comparisons, the new trajectory-aware preference method reduces ties to about 35%. This substantial reduction significantly enhances discriminative power, improves ranking stability, and boosts data efficiency, suggesting that perceived benchmark saturation might partly stem from inadequate evaluation measures.