FlashTrie Accelerates GPU-Based Constrained Beam Search for Retrieval.

Researchers have introduced FlashTrie, a GPU-accelerated system designed to significantly speed up constrained beam search in generative retrieval applications. This is particularly vital for scenarios like sponsored search, where generated document identifiers must precisely match a predefined library. Traditional CPU-based implementations of trie-constrained beam search face bottlenecks due to limited parallelism as beam width increases. FlashTrie overcomes this by employing an integer-aware succinct trie layout that compresses memory footprint, keeping the entire index in GPU high-bandwidth memory. It also features a cooperative CUDA kernel that handles beam expansion, validation, and pruning entirely on-device, eliminating per-step host orchestration. By replacing irregular CPU-style lookups with GPU-aware parallel primitives, FlashTrie improves warp utilization and reduces divergence. These innovations result in a dramatic reduction in decoding latency—under 3 ms for 800M keywords with beam widths up to 1000—and a 24x speedup over optimized CPU baselines. This enables up to 5x larger beam sizes in latency-critical applications, leading to a statistically significant +0.71% revenue lift in a large-scale online A/B experiment on a commercial search engine.