New Diffusion Models Enable Flexible, Any-Order Code and Text Generation

Many complex reasoning tasks, such as writing code or solving intricate problems, don't follow a strict left-to-right, causal sequence. Instead, humans often jump between high-level structures and fine-grained details, a process termed "any-order inference." While autoregressive language models struggle with this non-causal nature, masked diffusion models offer a promising "any-order" training objective and prediction interface. However, the research highlights a critical "interface-inference gap" in these models, primarily due to positional uncertainty. Fixed-canvas, token-level models might know what to generate but not where to place it. To resolve this, two complementary strategies are introduced: insertion-based masked diffusion, which allows generation across non-contiguous regions by relaxing fixed-position commitments, and latent-space masked diffusion, which shifts prediction to coarser semantic segments, enabling search over latent generation orders. These innovations lead to improved performance in tasks like Python coding and mathematical reasoning.