Spectral-LSH Compresses LLM Prompts Sub-Quadratically for Efficiency

Researchers have introduced Spectral-LSH, a novel, training-free method designed to compress long prompts for Large Language Models (LLMs) before they enter the model. This technique aims to mitigate the computational expense of prefill attention, which typically scales quadratically with sequence length. Spectral-LSH achieves sub-quadratic scaling by approximating the most influential components of an implicit attention-kernel operator. It does this by combining a Krylov subspace method with random features, thereby avoiding the need to explicitly materialize the computationally intensive O(N^2) attention kernel. Following this approximation, the method applies SimHash within the resulting attention eigenspace to identify and group similar tokens. These grouped tokens are then aggregated into "macro-tokens," which are assigned causal positional information. Experiments conducted on various Mistral and Qwen models using the C4 dataset revealed a distinct phase transition in compression effectiveness. While lightweight chunking is sufficient for low compression ratios (e.g., 4x), Spectral-LSH significantly preserves quality at higher compression rates (e.g., 8x and 16x), where chunking performance degrades. An adaptive backend combines both approaches, utilizing chunking for low compression and spectral clustering for high compression, though chunking remains faster overall for total latency.