Marginal Matching Fails to Ensure Style-Class Independence in Generative Models

This paper critiques a common assumption in factorized generative models: that matching the marginal distribution of a latent style variable to a fixed Gaussian prior ensures its independence from class information. The authors demonstrate that this interpretation is flawed. They show that marginal matching alone does not constrain class-conditional distributions, allowing the latent style to remain highly predictive of class labels despite appearing perfectly Gaussian in aggregate. The research provides an exact decomposition outlining four conditions necessary for factorized sampling, highlighting that the marginal mismatch is a critical failure point. Empirical studies on various models and datasets reveal that even with near-zero global MMD, linear probes can recover class labels from the style latent with high accuracy (74%-100%). Several mitigation strategies were tested, reducing probe accuracy but not fully eliminating within-class dependence. This work concludes that marginal distribution statistics alone are insufficient to certify independence from class labels, challenging a widely held belief in the field.