Activation Patching Flaws Revealed: Hidden Interaction Effects Impact Interpretability

A new research paper highlights a significant limitation in activation patching, a widely used technique for understanding the internal workings of AI models, particularly large language models. The study re-derives the activation patching estimand from causal mediation analysis and discovers that the Natural Indirect Effect (NIE), which is supposed to isolate the causal responsibility of individual model components, also implicitly includes "interaction effects" (INT). These INTs quantify how much a component's causal influence is contingent on the state of other components within the model. The presence of these hidden interaction effects means that activation patching may not accurately reflect the true causal importance of individual components. The paper demonstrates that components whose importance is conditional on others can either be overlooked entirely or have their significance artificially exaggerated. This phenomenon also accounts for the previously observed instability in faithfulness scores, which are metrics used to evaluate the reliability of interpretability methods. While attempts to eliminate INTs through estimator adjustments or changes in the unit of analysis prove problematic, the research suggests that INTs are not merely a nuisance. Instead, their magnitude and sign can serve as a diagnostic tool, indicating when causal conclusions are prompt-dependent or when a greedy, NIE-based ranking of components might miss crucial mechanisms that only combinatorial search could uncover. INTs scale with the distance between clean and patched activations and are negligible only when the model is locally affine.