LLM Interpretability Evidence Fails Analytic Variation Test

This paper critically examines the robustness of circuit-level interpretability evidence for Large Language Models (LLMs), particularly in the context of regulatory requirements like the EU AI Act. The Act mandates technical documentation describing how high-risk AI systems make decisions, with mechanistic interpretability being a primary source for such evidence. The core question investigated is whether this evidence remains stable when subjected to defensible variations in analytical settings. Researchers pre-registered a comprehensive grid of seven analytical axes, each level derived from published implementations, and mapped discovered circuits to structured Annex IV statements. Across 15,840 specifications on GPT-2 small for the indirect object identification task, the derived explanatory statement flipped in 73.2% of specification pairs. Even after standardizing the most influential choice (evaluation metric), the flip rate remained high at 59.4%. Crucially, the underlying circuits for these varying claims were found to be structurally near-disjoint (median pairwise Jaccard overlap of 4%) and functionally uncorrelated (Cohen's kappa 0.015). This instability indicates that the issue is not merely different descriptions of the same mechanism, but rather fundamentally different mechanisms being identified. The evidence consistently failed a proposed "filability criterion," suggesting that current mechanistic interpretability methods may not provide sufficiently stable or reliable evidence for regulatory compliance.