New Benchmark for Multi-Agent Routing in LLMs

A new research paper presents a benchmark designed to evaluate multi-agent routing for large language models (LLMs), framing it as a set-valued prediction challenge. In scenarios where a natural language query might require multiple AI agents, selecting the right set of agents while managing execution costs is crucial. The benchmark, derived from WildChat, comprises 3,000 prompts and a fixed catalog of 12 agents, with AI-assisted heuristic labels. It features a comprehensive evaluation protocol that combines set-level metrics (like Precision, Recall, F1), latency, a capability-coverage simulation, and a cost-aware routing setting. The study compared various methods, including nearest-neighbor matching, linear multilabel classification, fine-tuned encoders, and zero-shot LLM baselines. The findings indicate that supervised routing methods substantially outperform both nearest-neighbor and zero-shot LLM approaches. Specifically, fine-tuned encoders demonstrated the highest unconstrained set accuracy, while a linear multilabel model provided a strong practical baseline. When cost constraints were introduced, a weighted routing layer, particularly when applied on top of strong supervised scorers, significantly improved utility. This benchmark and evaluation protocol offer a reproducible framework for studying the trade-offs between accuracy and cost in multi-agent routing systems.