Set Diffusion Offers Faster, Flexible Language Model Decoding

Discrete diffusion models have shown quality improvements over autoregressive (AR) models but typically suffer from limitations like fixed-length generation and lack of key-value (KV) caching. While block diffusion partially addresses this by generating tokens in sequential blocks, it still restricts decoding flexibility and parallelism. This new work presents "Set Diffusion," a novel class of language models designed to bridge this gap. Set Diffusion features a likelihood parameterization that factorizes over flexible-position, flexible-length token sets, and a set-causal diffusion architecture that supports KV cache updates at every inference step. This allows tokens to be decoded in any order, including sliding-window sets, leading to faster inference and greater flexibility. The model demonstrates better speed-quality tradeoffs in tasks like mathematical reasoning, summarization, and unconditional generation compared to previous diffusion language models, and also offers stronger infilling capabilities than block diffusion. The code, model weights, and a blog post are available.