hLLM Achieves 64x Faster Generative Reranking with Single-Pass Decoding

Large language models (LLMs) are highly effective for generative ranking, but their autoregressive decoding process, which requires a sequential forward pass for each token, creates a significant efficiency bottleneck. This research identifies that for ranking tasks, the LLM only needs to output N ordinal values representing the ranked items, a highly structured output format. To address this, the paper introduces hLLM (Hungarian LLM), a specialized decoding strategy that decodes all N ordinals in a single forward pass, achieving O(1) complexity. hLLM works by extracting an N x K item-position score matrix from the LLM's prefill hidden states using a lightweight self-attention head. The optimal permutation (ranked order) is then determined by applying the Hungarian algorithm to this matrix, ensuring a valid ranking by design. Through systematic studies, the researchers demonstrate that fine-tuning with LoRA and teacher ranking distillation allows hLLM to achieve an end-to-end inference speed of 28 ms, representing a 64x speed-up compared to traditional methods, all while preserving the ranking quality of the teacher model. This innovation connects generative ranking with combinatorial optimization, opening new avenues for real-time ranking systems.