Probes Effectively Detect Tool-Calling Errors in Large Language Models

Large Language Models (LLMs) are increasingly integrated with external tools, but ensuring their correct and safe interaction is a growing concern. This paper investigates a novel method for identifying errors in how LLMs call these tools, by analyzing the "hidden states" within the models themselves. These internal states often contain rich information not evident from just looking at inputs and outputs. The researchers employed linear probes to detect tool-calling errors across 18 different LLMs. They discovered that this probing technique is highly effective, even catching subtle errors where an argument has the correct data type but an incorrect value – a type of mistake often overlooked by conventional logging systems. Key factors influencing the success of probing include the model's size, the specific layer being probed, and the type of post-training applied to the model. Importantly, the probes demonstrated an ability to generalize and detect new, unseen error types, which is crucial for real-world deployments.