LM Agents Show Promise in Explaining AI Model Circuits

Mechanistic interpretability has advanced significantly in automatically identifying specific "circuits" within AI models. However, the process of explaining what these localized components actually do remains a labor-intensive and difficult task to standardize. This research explores the potential of language model (LM) agents to aid in this explanation process once a circuit has been identified. The study introduces AgenticInterpBench, a new benchmark specifically designed for circuit explanation, comprising 84 semi-synthetic transformer circuits with 163 component-level annotations. Alongside this, the researchers propose HyVE (Hypothesize, Validate, Explain), an agentic explainer that iteratively analyzes each component. This process involves observing, generating hypotheses, and causally validating them, ultimately producing component-level explanations and a circuit-level task description. Across various LM backbones, HyVE successfully recovered valuable explanations for both components and tasks. However, no single backbone consistently outperformed others. The analysis revealed that while strong LMs typically form hypotheses grounded in observations, failures often occur later in the validation phase due to incomplete plans, code execution errors, or unresolved hypotheses. A case study on a Llama-3-8B arithmetic circuit suggests this approach can extend beyond synthetic benchmarks to naturally trained models, though robust validation remains a critical hurdle.