CLIP Latent Space Modeled as Hyperspherical Semantic Mixture

CLIP (Contrastive Language-Image Pretraining) models generate powerful semantic embeddings where similarity is measured by cosine distance, implying an inherent hyperspherical geometry. However, traditional probabilistic interpretations often rely on Gaussian assumptions, which fail to accurately represent this directional and multimodal structure. This research introduces a more principled density model for the CLIP latent space. The proposed model is based on Mixtures of von Mises-Fisher (MovMF) distributions, which are naturally defined on the unit hypersphere. Using the Expectation-Maximization (EM) algorithm, the researchers efficiently learn a probabilistic model where each mixture component corresponds to a distinct and coherent semantic concept. This formulation provides a closed-form likelihood that aligns perfectly with the hyperspherical geometry, enabling more accurate and interpretable density estimation. Empirical results demonstrate that this MovMF model significantly enhances performance in long-tailed and out-of-distribution detection tasks. Furthermore, it offers a natural way to decompose semantic embeddings, representing each as a sparse probabilistic combination of interpretable concepts. These findings suggest that the CLIP latent space is better characterized as a hyperspherical semantic mixture rather than an isotropic Gaussian, establishing a geometrically consistent framework for understanding multimodal representations.