STAR Improves Text-to-Image Generation with Adaptive Reward Allocation

Current reinforcement learning (RL) post-training techniques for text-to-image generation typically convert the final image's quality into a single numerical reward, which is then applied uniformly across the entire image generation process. This approach overlooks the inherent temporal and spatial structure of image generation, where different denoising steps contribute to distinct stages of the image, and specific parts of the image are more critical for aligning with the input text. This mismatch in granularity hinders policy updates from effectively targeting the most impactful generative components. To overcome this limitation, researchers have introduced SpatioTemporal Adaptive Reward (STAR) Allocation. This method is designed for RL post-training of text-to-image diffusion and flow models. STAR leverages the internal text-image attention mechanisms of the generative model, focusing on the key content specified in the user's prompt. STAR dynamically creates spatial allocation maps that evolve across different denoising steps and generation rollouts. It then assigns a group-relative advantage to the most relevant latent regions of the image, incurring minimal additional computational overhead. By applying stronger policy updates to these specific, spatially resolved regions through a tailored objective, STAR effectively guides the model. Evaluated on Stable Diffusion 3.5 Medium across tasks like GenEval, OCR text rendering, and PickScore, STAR demonstrated significant improvements in compositional semantic alignment, text rendering accuracy, and overall preference optimization, all without altering the external reward source.