Rational Sparse Autoencoder Improves Mechanistic Interpretability.

Sparse autoencoders (SAEs) are fundamental tools for mechanistic interpretability in large language models, helping to understand how models represent information. However, existing SAE families are limited by fixed encoder nonlinearities, such as ReLU or TopK, which can constrain the model's ability to achieve an optimal balance between reconstruction quality and sparsity. This fixed nature can also distort the trade-off between these two critical aspects. The Rational Sparse Autoencoder (RSAE) addresses this limitation by replacing the fixed encoder activation with a trainable rational function. Rational functions are highly flexible and can approximate the activation primitives used by current SAE families, while also offering a richer function class to adapt to the specific geometry of pre-activations. The RSAE implementation involves a two-stage pipeline: an initialization step that copies baseline SAE weights and calibrates rational coefficients, followed by a fine-tuning phase using the standard sparsity-regularized reconstruction objective. Empirically, RSAE consistently improved upon baseline SAEs across various open-weight language models and activation families. It showed gains in both reconstruction metrics and downstream behavior metrics, all without compromising feature-level interpretability. These improvements were observed across different host models and sparsity levels, with the upgrade requiring minimal additional parameters and computational resources.