Source-Grounded Feature Inversion for Neural Network Interpretation

Interpreting the internal workings of neural networks, particularly understanding what specific features extract from an input, is a significant challenge. Existing feature inversion techniques aim to reconstruct an input that activates a target feature, but they often produce generic results because many inputs can satisfy the target activation. This paper proposes a new approach called source-grounded feature inversion, which addresses this by conditioning the inversion process on the local network geometry of the *original* input that generated the feature. The method leverages backpropagation but refines its signal using a closed-form matrix Wiener map to accurately estimate upstream states. This allows for a single, finite reverse pass through the computational graph to reconstruct the source-grounded inverse. The technique is robust across various CNN and Transformer architectures and visual distributions, producing visualizations that directly link the network's internal evidence to its decision-making process, enabling detailed inspection of hidden feature hierarchies.