Information Theory Explains Generalization in Bayesian Diffusion Models

A fundamental mystery in diffusion models is how they learn complex distributions and generalize from finite data without merely memorizing it. This research addresses this by introducing analytically tractable Bayesian Information Restricted Diffusion (BIRD) models. These models infer past training samples by observing restricted information about noisy data, generalizing existing analytical diffusion models that use spatially local information restriction. The study reveals an exact information-theoretic phase boundary that dictates whether a BIRD model memorizes or generalizes. Generalization occurs when the mutual information between its restricted noisy observations and the training data falls below the logarithm of the number of training points. Experiments confirm this theoretical prediction, showing that both BIRD models and early-training diffusion models operate near this boundary, increasingly restricting information over time. This highlights information restriction as a key mechanism for circumventing the curse of dimensionality in generative AI.