Research Explores Sparsity and Superposition in Autoencoder Loss

A significant challenge in understanding how neural networks function, known as mechanistic interpretability, is the issue of polysemanticity. This occurs when individual neurons are involved in multiple tasks, making their specific role difficult to pinpoint. A key theory suggests this phenomenon arises from "superposition," where neural networks represent different features using non-orthogonal directions within a lower-dimensional space. This strategy allows for efficient data compression without sacrificing accuracy, especially when input vectors are sparse. This new work provides a mathematical foundation for these observations, rigorously analyzing the occurrence and optimality of superposition. It offers upper and lower bounds for L2 reconstruction loss, particularly tight in highly sparse conditions, for autoencoders using power activation functions, thereby validating earlier empirical findings.