New Hybrid Diffusion LLM Boosts Long-Context Generation Throughput

Generating long sequences with large language models (LLMs) efficiently is a major challenge, primarily due to memory bandwidth limitations from the growing Key/Value (KV) cache. Current solutions either focus on reducing memory access through efficient attention or linear mixers like Mamba, or by generating tokens in parallel blocks. Combining these two approaches has proven difficult. Previous hybrid diffusion models, such as DiffuMamba, used bidirectional Mamba mixing, which prevented efficient caching across blocks because the reverse scan required processing the entire sequence. This new research introduces a BDLM Mamba-attention hybrid that solves this by restricting the reverse Mamba scan to only the active denoising block. This crucial modification allows for exact caching across blocks, dramatically improving efficiency. The proposed BDLM Mamba-H model demonstrates superior performance, achieving better perplexity on C4-en validation compared to other BDLM variants at 87M parameters. More importantly, for long-context inference, it delivers up to 19.7 times the throughput of full-sequence DiffuMamba-H at 65K tokens and 3.7 times that of BDLM attention at 262K tokens, positioning Mamba hybrids as a promising architecture for high-throughput, long-context generation.