D2PO Optimizes Diffusion Samplers with Dynamic Preference Alignment

This research introduces D2PO, a novel framework designed to enhance the quality of images generated by diffusion models, particularly when using fewer sampling steps. Current methods often struggle to maintain fine-grained details, leading to a mismatch between the generated output and human perception. D2PO tackles this by reframing the optimization problem as a preference-based alignment, drawing inspiration from the Direct Preference Optimization (DPO) framework. The core innovation involves modeling the sampling policy as an energy-based model, which allows for the comparison of preferences to be translated into measurable energy differences. A key aspect is the introduction of dynamic preferences, where the preferred samples used for training progressively improve as the model learns. This iterative self-improvement mechanism replaces static teacher supervision, leading to more faithful alignment with desired perceptual quality. Experiments show that D2PO consistently outperforms traditional regression-based schedulers, especially under low sampling step constraints, by unlocking the full potential of high-quality teachers and better preserving both structural consistency and fine details.