New Method Improves Diffusion Language Model Accuracy at Test Time

Diffusion Language Models (DLMs) are increasingly competitive with traditional autoregressive models, sometimes even surpassing them in performance. Unlike autoregressive models that generate text sequentially from left to right, DLMs produce output through an iterative denoising process without a fixed order. To further enhance DLM performance, a new test-time self-verification technique called Prefix-Denoising Consistency (PDC) has been developed. PDC capitalizes on a unique characteristic of DLMs: when a sentence is split and the latter part is regenerated conditioned on the fixed prefix, correct outputs tend to be more stable and reproducible than incorrect ones. By leveraging this "prefix-conditioned regeneration" signal, PDC consistently improves the quality of initially generated samples. It has been shown to outperform independent generations under computational constraints and demonstrates robustness across different unmasking strategies and parameter configurations, particularly in mathematical and commonsense reasoning tasks.