Rollout Training Improves Constrained Diffusion Models

New research introduces a novel fine-tuning framework aimed at enhancing constrained generative models, specifically diffusion models. These models are designed to produce outputs that not only resemble real data but also adhere to specific, often complex, feasibility constraints. Existing methods typically enforce these constraints either during training or during the sampling (inference) phase, each with its drawbacks. Training-time optimization can suffer from a mismatch between training and sampling conditions, while sampling-time corrections often require extensive tuning and can introduce distribution shifts. The proposed framework addresses these limitations by integrating constraint guidance directly into the training process through "online rollout." This technique differentiates through the fixed noise schedule used in the denoising process, effectively exposing the model to constraint violations that might arise along the entire denoising trajectory. By aligning the learning process with the actual sampling process, the model learns to avoid constraint violations more effectively. Experiments conducted across various tasks demonstrate that this rollout-based training significantly improves constraint satisfaction in generated samples. Crucially, it achieves this while maintaining competitive sampling quality compared to previous methods.