New Method Enables Constrained Decoding for Diffusion Language Models.

Constrained decoding is a critical requirement for Large Language Models (LLMs), ensuring that their generated outputs conform to specific formats, such as JSON schema for function calls. While existing systems are designed for autoregressive models that generate text left-to-right, diffusion language models present a challenge because they sample multiple positions simultaneously. Researchers have developed an exact and tractable algorithm to enable constrained decoding for diffusion language models. This approach samples from a constrained mean-field posterior under any constraint expressible as a finite automaton. By treating finite automata as graphical models, the method achieves tractable representations of the constrained distribution, guaranteeing constraint satisfaction. The algorithm supports both greedy and sampling-based decoding, is compatible with parallel and block-wise decoding, and uses depth-reduction techniques to reduce sampling depth. Empirical evaluations on models like Dream-7B and LLaDA-8B show substantial accuracy gains in tasks such as function calling, planning, text-to-SQL, and math reasoning, with minimal inference overhead. For instance, Dream-7B's greedy decoding accuracy on BFCL-Live improved from 63.9% to 71.5%.