Canary Tools Diagnose LLM Agent Tool-Selection Weaknesses.

Evaluating large language model (LLM) agents often reveals *that* a model chose the wrong tool, but rarely *why*. This study addresses this gap by introducing "canary tools," which are specially designed diagnostic probe tools inserted into an agent's Model Context Protocol (MCP) tool set. Each canary tool is engineered to expose a specific type of tool-selection weakness, categorized into a six-type taxonomy: semantic decoys, parameter traps, capability mirages, prerequisite blindness, temporal decoys, and granularity traps. This innovative approach transforms a simple "wrong tool" outcome into a multi-dimensional profile of an agent's reasoning about tools. The researchers conducted extensive evaluations across eight models of varying capabilities, performing thousands of runs. Key findings include a dramatic difference in susceptibility across models, with the most capable models showing significantly lower rates of being trapped by canary tools. Interestingly, capability tier alone does not perfectly predict safety; some mid-tier or even cheaper models from the same provider demonstrated higher robustness. The taxonomy itself proved capability-stratified, with "capability mirages" effectively trapping frontier models, while other types were more effective against smaller, open models. The study confirms that these probes measure reasoning rather than mere phrase-spotting, and susceptibility to canaries correlates with overall task failure.