LLM Routers Show Limited Task-Specific Advantage in Benchmarks

In agentic systems, model selection is increasingly delegated to routers, but direct comparisons of open-source routers have been difficult due to varied evaluation protocols. A new study introduced a common measurement protocol and hybrid evaluation for four open-source router implementations across four benchmarks: RouterBench, BFCL v4, tau2-bench, and WebArena. The evaluation involved 290 frozen tasks against a locked matrix of 2,610 candidate outcomes. The findings revealed that three of the four routers consistently assigned models to constant or near-constant tiers, showing little dynamic adaptation. Only the vLLM Semantic Router exhibited material variation based on prompt content, yet it did not achieve the highest success rate on any of the benchmarks. Intriguingly, a simple "Always-Mid" baseline matched one router's performance almost exactly. The study concluded that observed performance gains were more closely tied to the composition of the selected model tier rather than the router's ability to target specific tasks effectively. This emphasizes the necessity of including fixed-tier baselines and selected-tier distributions as controls in future router evaluations.