InferenceBench Evaluates AI Agents for LLM Optimization

A new benchmark named InferenceBench has been introduced to assess the capabilities of AI agents in optimizing large language model (LLM) inference in an open-ended setting. Unlike traditional benchmarks with prescribed workflows, InferenceBench tasks agents with deploying an OpenAI-compatible inference server and optimizing its speed on an H100 GPU within a two-hour time limit. The benchmark features four distinct optimization scenarios, each targeting different bottlenecks like prefill latency, decode latency, or concurrent request throughput. Across 15 frontier agent configurations, agents consistently outperformed a naive PyTorch baseline, achieving up to 8.08x speedup, and often matched or exceeded default settings of serving engines like vLLM (4.05x). However, they still fell short of a simple hyperparameter search conducted within the same time budget (up to 11.53x). Qualitative analysis revealed that agents, despite enumerating many relevant optimization techniques, tended to converge on a single inference framework, testing only a few configurations and spending significant time on re-measurement or repair rather than exploring diverse strategies. This suggests that the primary bottleneck for agents is not domain knowledge, but rather the ability to propose varied configurations and systematically evaluate them.