JEPA World Models Plan Effectively with Point Cloud Data

Joint Embedding Predictive Architecture (JEPA) world models have proven effective for latent-space planning, offering a practical route to control in various applications. However, these models have almost exclusively relied on image-based observations. A critical question has been whether latent prediction can effectively operate with geometric observations, specifically point clouds, which are inherently sparse, unordered, and prone to self-occlusion. This new research lifts three canonical JEPA designs—frozen-encoder, distribution-prior, and action-sensitive—to handle point cloud data. By re-sensing a stable-worldmodel benchmark to only differ in observation type from image baselines, the study found that all three models successfully plan without collapse. The distribution-prior model achieved statistical equivalence to its image counterpart, and the action-sensitive model performed strongest when significant geometry was in motion. Probing revealed that object positions are highly linearly decodable, and attention focuses on moving points. The models also demonstrated robustness to heavy dropout and enabled a natural 3D target goal interface without requiring a goal observation.