Cosine-Scored Sparse Autoencoders Improve Feature Learning in AI Models

Sparse autoencoders (SAEs) are used to detect features in AI models, typically relying on an inner product score. This standard method can lead to issues where high-norm input tokens disproportionately activate features, regardless of their actual content alignment, especially after sublayer normalization has removed magnitude information. Researchers propose a novel scoring mechanism for SAEs, replacing the inner product with a learned blend of cosine similarity and input magnitude. This allows the optimizer to determine the optimal balance, and surprisingly, features never choose more than half-magnitude dependence, indicating a preference for content over raw magnitude. The new cosine-scored autoencoders, at matched reconstruction performance, learn features that align significantly more often with human-recognizable concepts. This suggests that the traditional inner product wastes dictionary slots on "norm detectors," while the cosine-based approach fosters more meaningful and interpretable feature representations.