New PRQ-KMeans Improves Semantic ID Tokenization for Generative Retrieval.

Semantic identifiers (SIDs) are crucial for modern generative retrieval and recommendation systems, representing entities as hierarchical token sequences. Current residual-quantization tokenizers build these sequences by iteratively selecting codewords and passing residuals. However, this approach has limitations, such as first-level capacity being consumed by corpus-wide shared components, rigid codeword assignments, and incomplete residual subtraction. A new method, PRQ-KMeans, tackles these issues by operating in a post-hoc setting where residual construction is not tied to input reconstruction. It first removes the global mean, then refines centroids using Top-k similarity-weighted updates, and finally replaces full-codeword subtraction with a projection residual. This projection specifically removes the selected-centroid component from each representation. Evaluations on a large industrial search dataset and four public recommendation benchmarks demonstrate that PRQ-KMeans outperforms other tokenizers. It achieved significant gains, including up to 7.4% in HitRate and 11.8% in MRR on the industrial dataset, indicating its effectiveness in improving semantic ID tokenization.