D2PO Optimizes Diffusion Samplers with Dynamic Preference

Current methods for optimizing diffusion samplers, particularly student-teacher regression frameworks, often train low-NFE (Number of Function Evaluations) student samplers to mimic high-NFE teachers. This can lead to a loss of high-frequency texture fidelity, even if coarse global structures are preserved, resulting in a misalignment with desired perceptual quality. To address this, researchers propose D2PO (Dynamic Direct Preference Optimization), a principled framework that redefines sampler optimization as a preference-based alignment problem, leveraging the Direct Preference Optimization (DPO) framework. D2PO models the sampling policy as an energy-based model (EBM), translating preference comparisons into manageable energy differences. A key innovation is a novel energy formulation derived directly from the pretrained score network, enabling preference evaluation in perturbed spaces that capture both structural consistency and fine-grained details. Furthermore, D2PO introduces dynamic preferences, where the preferred samples used for alignment progressively improve as the sampling policies are learned, offering a self-improving refinement process that surpasses rigid static teacher supervision.