LinkedIn Boosts Semantic Search with Efficient GPU Retrieval

LinkedIn has developed a new, highly efficient GPU retrieval framework designed to enhance semantic search capabilities, particularly for complex natural-language queries like "a fintech founder in Berlin who worked in payments." Traditional cosine similarity often averages evidence, potentially masking a failure on one critical facet with a strong match on another, thereby limiting recall. To address this, LinkedIn's new framework aligns with a bottleneck-oriented relevance policy, where every non-negotiable facet must be satisfied. The core of this framework involves partitioning embeddings into eight category-supervised segments. Scores for these segments follow a min/median rule during serving, and for multi-vector retrieval, the segment score is computed independently per tagged document slot and maximized across slots. A lightweight single-slot Stage-1 scorer generates high-recall candidates, while scale-invariant relative-norm gating ensures consistent category activation. This framework is served using a two-stage GPU architecture. An FP8 coarse ranker efficiently scores the entire corpus, boosting per-shard capacity by 71% and Stage-1 matmul throughput by 36%. Subsequently, an FP16 stage precisely re-ranks an oversampled candidate set, recovering nearly all full-FP16 recall at over 500 queries per second per shard replica. Offline evaluations on 21,000 held-out queries showed improved relevance, and a live A/B test demonstrated significant gains in exploratory-query Precision@10 and navigational Precision@1, confirmed by human evaluation.