Lambda-VAE Solves Posterior Collapse in Variational Autoencoders

Variational Autoencoders (VAEs) frequently encounter a significant issue known as posterior collapse, where the approximate posterior distribution converges to the prior, rendering the latent code uninformative. Despite extensive research, a comprehensive understanding of its root causes has remained elusive. This new study identifies and formalizes two distinct yet interconnected causes: gradient imbalance, where the decoder's reconstruction signal diminishes faster than the KL regularization, and information gap, where stochastic sampling discards too much of the encoder's representation. The paper introduces $\lambda$-VAE, a solution that tackles both issues with a single modification to the reparameterization step. By scaling the sampling noise with a per-dimension exponent while retaining the original posterior variance for the KL penalty, $\lambda$-VAE shifts the training attractor away from the collapsed state, promoting "variance equalization" across all latent dimensions. This method demonstrates consistent reductions in collapsed dimensions, significant information capacity gains, and improved reconstruction quality across standard benchmarks.