XP-JEPA Grounds Latent Dynamics in Physics for Better Control

In self-predictive models, the encoder and predictor are often optimized together, which can lead to latent transitions that are easy to predict but not strongly tied to the actual physical evolution of a scene. To address this, researchers have introduced the cross-predictive JEPA (XP-JEPA). This model aims to ground visual latent dynamics by incorporating privileged physical trajectories during training. XP-JEPA operates by separately encoding visual observations and physical states. Both of these are then advanced through a shared predictor that is conditioned on actions. The model's objective is to match each prediction to both future representations, thereby encouraging unified latent dynamics across the visual and physical modalities, firmly rooted in the underlying physical transitions. Crucially, the physical branch is only used during training and is discarded at deployment, leaving a visual-only model. This approach significantly reduces rollout drift of a newly fitted predictor from 0.361 to 0.104 and boosts mean control success from 53.6% to 78.2% across a multi-task suite. This demonstrates that cross-predictive physical grounding can yield more forecastable latent dynamics for rollout-based control without needing physical inputs during testing.