Sparse Autoencoders Enhance Interpretability and Control of Sentence Embeddings

Dense sentence embeddings are foundational to modern Retrieval-Augmented Generation (RAG) systems, yet their inherent opacity, caused by feature superposition, limits interpretability. This lack of transparency makes it difficult to align retrieval processes with human intent, as the entangled representations are challenging to analyze or control directly. This research introduces a novel method to address this by disentangling the dense representations produced by sentence transformers, such as E5, into human-interpretable concepts. This is achieved through the application of Top-k Sparse Autoencoders (SAEs). The study demonstrates that these disentangled features correspond to specific semantic, syntactic, and pragmatic categories. Furthermore, the paper presents an activation steering mechanism that enables precise intervention in the retrieval process. By clamping specific latent features, it becomes possible to re-rank search results to better match user constraints, all without the need to retrain the underlying backbone model. These findings suggest that SAE-based decomposition offers a promising pathway towards more transparent and steerable neural information retrieval systems.