NodeJEPA Improves Graph Self-Supervised Learning with Latent Prediction

Self-supervised learning on graphs typically relies on contrastive methods with specific augmentations or generative methods that reconstruct input attributes. Both approaches can inadvertently tie learned representations to low-level input statistics rather than the underlying relational structure of the graph. NodeJEPA, a new joint-embedding predictive architecture, offers an alternative by learning through the prediction of latent targets. NodeJEPA is designed for node-level tasks and operates by masking structure-aware k-hop ego-subgraphs. A context encoder then predicts the latent representations of these masked nodes, with targets provided by an EMA-updated target encoder. A key innovation is the structure-conditioned predictor, which incorporates spectral and centrality descriptors via cross-attention, ensuring the model focuses on structural signals. The framework also employs regularization techniques like variance, covariance, and Laplacian spectral regularizers to stabilize the embedding geometry, along with an optional curriculum for gradually increasing masking difficulty. By predicting in latent space, NodeJEPA bypasses the need for input reconstruction or manual graph augmentations, leading to more robust and structurally informed node representations, as demonstrated on standard node classification benchmarks.