New Framework Analyzes LLM Agent Tool Discovery and Use

The Lomekwi framework introduces a novel way to analyze how Large Language Model (LLM) agents interact with tools. Instead of just measuring overall task success, it separates the act of using a tool from the more complex process of discovering it. Tool discovery is further broken down into three distinct components: curiosity, which is the agent's ability to identify necessary parts; recognition, its capacity to understand how to construct the tool; and efficiency, its skill in applying the tool once created. This new decomposition can be applied to existing benchmarks, such as Voyager, providing a more granular understanding of agent capabilities. A key finding from the research is an inverse scaling relationship for recognition, meaning that larger models surprisingly show reduced ability in this specific aspect of tool discovery. This observation was consistent across combinatorial games and environments designed to mimic real-world scenarios, suggesting a fundamental challenge in how LLMs currently handle tool recognition.