Diffusion Models Accurately Recover Mixture Weights Despite Score Insensitivity

Score-based generative models often exhibit a perplexing behavior: they can generate samples that cover all modes of a multimodal distribution, yet sometimes fail to accurately represent the relative proportions (mixture weights) of these modes. This research delves into this paradox, aiming to understand how these models can recover accurate mixture weights even when their underlying score functions appear insensitive to them. The study reveals that accurate recovery of mixture weights is possible if the scores at *intermediate* noise levels contain informative signals about these weights, even if the target score itself is insensitive. To quantify this, the researchers define the Diffusion Score Sensitivity Index (DSSI), which measures the variation in the Diffusion Score Matching (DSM) loss relative to changes in a parameter. DSSI is shown to govern how accurately a parameter of the target distribution can be estimated from generated samples. For Gaussian mixtures, the research theoretically proves that mixture weight estimation errors are proportional to the DSM loss under certain conditions. Empirically, the study demonstrates that sensitivity emerges during the noising process of benchmark data, and these sensitivity values predict how well a well-trained model recovers mixture weights. Furthermore, the choice of noise schedule can significantly impact diffusion sensitivity, potentially leading to mode amplification if not carefully managed.