Block-Sparse Featurizers Analyzed for Feature Learning

This research delves into the Block-Sparse Featurizer (BSF), a novel architecture akin to a sparse autoencoder but which uses small subspaces (blocks of directions) as its fundamental unit instead of single directions. BSFs are particularly suited for learning features that reside on low-dimensional manifolds, a common characteristic in computer vision tasks. The analysis reveals that despite its advantages, BSFs still encounter classic sparse autoencoder challenges such as "feature splitting" (where a single concept is represented by multiple features) and "feature composition" (where multiple concepts are conflated into one feature). To address these limitations, the paper proposes several architectural modifications, notably a "Tournament Top-K" selection rule. This new rule is shown to significantly reduce feature splitting, enhancing the interpretability and efficiency of the learned representations. The block paradigm is also extended to crosscoders, suggesting broader applicability.