Probabilistic JEPA Interpreted as Hidden Markov Model

Joint-Embedding Predictive Architectures (JEPAs) are a class of models focused on learning representations by predicting future or masked data from current observations. This research uncovers a fundamental connection between a specific type of JEPA, the full, time-indexed Predictive Information Bottleneck VJEPA (PIB-VJEPA), and Hidden Markov Models (HMMs). It demonstrates that PIB-VJEPA exhibits the same core computational roles as an HMM: inferring hidden states, propagating through dynamics, and emitting back to observation space. To solidify this interpretation, the paper introduces Markov-Chain JEPA (MCJEPA), which replaces the latent predictor with a learned transition matrix, ensuring consistency across multiple prediction horizons. This state-space perspective provides a principled understanding of temporal JEPA, where predictive information-bottleneck learning can be seen as seeking a compact and sufficient predictive state. Experimental results support this HMM-like filtering interpretation and the concept of predictive Markovization.