AV-JEPA Extends Self-Supervised Learning to Audio-Visual Data

The paper introduces AV-JEPA, a novel framework that extends the LeJEPA (Joint Embedding Predictive Architecture) concept to the realm of audio-visual self-supervised learning. This approach aims to learn robust representations from combined audio and video data without requiring explicit labels. AV-JEPA employs an early-fusion Vision Transformer and utilizes modality dropout as a masking strategy. The core training objective is to align the embeddings generated from global views with those from local, per-modality views. A key feature is the SIGReg objective, which encourages an optimal distribution in the latent space. The resulting architecture is notably streamlined, avoiding common complexities found in other self-supervised methods, such as decoders, EMA teachers, intricate multi-term losses, or contrastive negatives. This simplicity allows AV-JEPA to achieve strong classification performance on benchmarks like VGGSound and AudioSet, and it inherently supports zero-shot audio-video retrieval.