HARP: Training-Free Interpretability for Neural Networks with Retrieval.

Interpreting the internal workings of neural networks typically involves methods ranging from simple, training-free techniques like linear probes to more complex, training-intensive approaches such as SAEs (Sparse Autoencoders) and activation oracles. Training-based methods are often considered more powerful due to their reliance on large activation datasets. This raises a fundamental question: do these expensive methods truly uncover insights that are not already present or recoverable from the training data itself? To investigate this, researchers developed HARP (Hypothesis-driven Agentic Retrieval and Probing). HARP equips a large language model (LLM) agent with a vector database containing neural network activations paired with their textual contexts. The agent also has tools to manipulate these activations, such as projecting out directions, computing differences, and averaging. The HARP agent iteratively queries the database, formulates hypotheses from the retrieved samples, and then validates these hypotheses by constructing linear probes. Remarkably, despite being entirely training-free, HARP surpassed both activation oracles and SAE-based agents in several key interpretability tasks, including concept discovery, concept detection, model steering, and secret elicitation. Its training-free nature also makes HARP significantly more cost-effective and flexible, allowing new datasets to be indexed on demand. These findings suggest that current training-based interpretability methods may not yet extract insights beyond their training data, prompting a need for new benchmarks that explicitly demand such deeper insights.