ToolMenuBench Evaluates LLM Agent Tool-Menu Filtering Strategies

Large Language Model (LLM) agents are increasingly augmented with extensive tool libraries, but current evaluations often focus only on whether a model can correctly invoke a tool. There's a critical gap in understanding how the *visible* tool menu influences an agent's reliability, efficiency, and exposure to safety-relevant risks. To address this, researchers introduce ToolMenuBench, a benchmark specifically for evaluating tool-menu construction in multi-step LLM agents. ToolMenuBench systematically varies factors like tool-menu size, distractor types, state-dependent task structures, and risk exposure. It provides both filter-level metrics (e.g., visible-tool count, risky-tool exposure) and downstream agent metrics (e.g., task success, wrong-tool calls, token usage). In a controlled evaluation across diverse models and filtering methods, Causal Minimal Tool Filtering (CMTF) emerged as highly effective. CMTF improved task success from 32.1% (with all tools exposed) to 85.7% and reduced average token usage by approximately 98%. This method achieved the strongest overall trade-off, significantly reducing visible tools, incorrect tool calls, premature actions, and risky tool exposure compared to unfiltered, lexical, state-aware, and broader causal-path baselines. ToolMenuBench offers a reusable framework for tackling the "agent-interface problem" – determining which tools should be visible, when, and under what cost or risk constraints.