PreDiff-LM Improves Diffusion Models with Hybrid Attention

Discrete masked diffusion language models offer capabilities like bidirectional generation and infilling, but adapting them from pretrained autoregressive (AR) transformers presents a challenge: reconciling the causal nature of AR pretraining with the bidirectional requirements of denoising. This research addresses this by introducing PreDiff-LM, which employs a novel hybrid attention mechanism. This mechanism maintains causal attention within the observed prompt while enabling full bidirectional attention within the masked target segments. When compared to uniform bidirectional attention using the same AR initialization (a matched GPT-2 Medium setup), PreDiff-LM's hybrid mask significantly improved unconditional perplexity from 34.1 to 28.7 and MAUVE from 0.71 to 0.78. This attention adaptation also complements a DiffuGPT-style objective adaptation, further reducing perplexity to 26.9. While pretrained initialization drastically cut the steps needed to reach a perplexity below 50 (from 350K to 8K), optimized AR models still outperform at equal scale. Beyond perplexity, PreDiff-LM showed improvements in repetition, distributional quality, four zero-shot downstream tasks, and human preference over previous diffusion baselines, positioning hybrid attention as a valuable method for adapting pretrained causal backbones.