Semantic DLM+ Improves Diffusion Language Models with Bias-Variance Trade-off

Diffusion Language Models (DLMs) are a promising alternative to autoregressive models, but their performance is highly dependent on the design of their transition kernels. Poorly chosen kernels can lead to significant problems such as unstable training, slow convergence, and biased output generation. This research analyzes these sensitivities by examining generalization error, pinpointing asymptotic bias, exposure bias, and optimization variance as key factors. The study compares different transition kernels, noting that masking diffusion offers easier posterior approximation but uniform diffusion provides stronger sampling-side repair. Semantic DLM (SemDLM) is revisited as a potential middle ground, corrupting tokens to semantically similar neighborhoods. However, SemDLM was found to suffer from a "semantic basin problem," producing low-diversity text. To overcome this, the researchers propose SemDLM+, which incorporates a global transition mechanism and a semantic-frequency penalty during sampling. Experiments on large datasets demonstrate that SemDLM+ significantly improves training dynamics and achieves competitive language modeling and generation quality with enhanced diversity.