New Statistical Regularizers Enhance Self-Supervised Learning Stability and Convergence

This research introduces an advancement in self-supervised learning (SSL) by addressing a common issue in representation collapse prevention. Existing SSL frameworks often use sliced statistical regularizers that rely on Monte Carlo sampling, which introduces variance into training gradients, leading to unstable optimization and slower convergence. The new approach proposes analytically integrating out these random projections, resulting in deterministic Maximum Mean Discrepancy (MMD) objectives. This method bypasses the variance associated with sliced techniques. The study formulates full-dimensional objectives for MMD, Kernel Stein Discrepancy (KSD), and Kullback-Leibler (KL) divergence directly on the hypersphere, using rotationally invariant kernels to prevent spatial bias. Empirical results demonstrate that eliminating projection-induced noise leads to more stable optimization, faster convergence, and improved performance on datasets like ImageNet and Galaxy10. The choice of statistical test also influences the learned latent space geometry, with MMD and KSD favoring clustered organization and KL divergence promoting fine-grained instance separation.