Orthogonal JEPA Improves Latent World Models

Researchers have developed a novel framework called Orthogonal JEPA (Joint-embedding Predictive Architecture) to enhance the learning of latent world models. Traditional JEPAs often struggle with complex systems because their single, monolithic latent state can overemphasize dominant signals, leading to weaker or conflicting gradients for less prominent predictive structures. Orthogonal JEPA tackles this by factorizing target states into multiple orthogonal components. The framework employs learned basis matrices to analyze each target state, with a dedicated prediction branch estimating each component from a shared context representation. This approach ensures that predictive regression maintains factor magnitudes, an orthogonality objective prevents redundancy, and regularization techniques maintain variation and prevent encoder collapse. The synthesized latent state can then be effectively used for various downstream tasks like readouts, decoding, planning, or autoregressive rollouts, demonstrating improved representation quality and long-horizon stability across diverse applications.