Vocabulary Gap Hinders Advanced Encoders in Sparse Retrieval

While advanced foundation models like ModernBERT excel in dense retrieval tasks, they surprisingly fall short compared to older architectures such as BERT-base in learned sparse retrieval (LSR). This discrepancy has been attributed to a "Vocabulary Gap." Modern tokenizers often use raw, case-sensitive vocabularies designed for lossless reconstruction, which can map single semantic units to multiple surface forms. This redundancy wastes model capacity and impedes effective lexical matching. The researchers formalized this concept, demonstrating that a more coarse-grained vocabulary, if semantic integrity is maintained, can improve generalization by simplifying the hypothesis class. To address this, they introduced Vocabulary Transfer (VT), a framework that allows advanced encoders to adopt sparse-friendly, normalized vocabularies with minimal computational overhead. VT employs Semantic Initialization to preserve geometric structure and Activation Potential Calibration (APC) to align pre-trained manifolds with sparsity constraints, preventing issues like dead neurons. Empirically, VT significantly boosts performance, enabling ModernBERT to achieve state-of-the-art results on the BEIR benchmark and revitalizing underperforming models like RoBERTa-large, confirming that the issue is a solvable vocabulary mismatch rather than an architectural flaw.