PAPA Optimizes Diffusion Models with Real-Time User Feedback

Diffusion models have proven highly effective in generating complex data, but their application in areas like personalized recommender systems requires aligning them with evolving user preferences. Traditionally, this involves fine-tuning the model based on a reward function, which in turn necessitates a large, parameterized reward model built from extensive preference data—a resource often unavailable in real-world scenarios. This research introduces PAPA, or Personalized Active Preference Alignment, to overcome this limitation. PAPA directly optimizes the diffusion model using real-time user feedback, eliminating the need for a separate, pre-trained reward model. Inspired by variational inference, this approach enables highly feedback-efficient preference alignment. The paper demonstrates PAPA's effectiveness through various experiments, including class-conditioned and fine-grained alignment tasks. Furthermore, theoretical insights led to an enhanced strategy, EPAPA, which reduces computational costs and accelerates the fine-tuning process, making it even more suitable for practical deployment.