LLM Agent Routers Misjudged by Static Replay Evaluation

This paper critically examines the common practice of evaluating LLM routers, particularly those used within multi-step agents, through static replay. Current methods typically replay logged trajectories and simply substitute a different model's recorded outputs, assuming the rest of the agent's path remains unchanged. The researchers challenge this assumption, arguing that such static evaluation "scores the wrong world." To test this, they employed "branching rollouts," where live agent trajectories (specifically on SWE-bench) were forked at controlled points. The environment was rebuilt, and each fork continued with a different model, allowing for a direct comparison against control forks that isolated sampling noise. The findings were stark: model swaps led to significant divergence, rewriting 61-94% of post-fork actions, far exceeding the divergence seen in control runs. A high percentage of early swaps diverged at the very first action after the fork, invalidating most replayed states. Crucially, all observed outcome flips (e.g., an unsolved instance becoming solved or vice versa) occurred in the swap arms, never in the control forks. When these same swaps were scored using a log-stitching replay evaluator, the replay method consistently mispredicted success-relevant outcomes and produced patches with extremely low similarity to actual reality. The study also uncovered that "temperature-0 determinism" is configuration-dependent, with different serving methods (e.g., FP8 vs. AWQ) leading to vastly different levels of divergence. This research concludes that replay-based benchmarks are unsuitable for evaluating agentic routing and provides a harness and trajectories for future, more accurate evaluations.