Graph Convolutional Attention Improves Graph Denoising and Diffusion

Denoising graphs is a foundational task in graph learning and a core component of modern graph diffusion models. While attention-based architectures, such as graph transformers, have shown considerable promise in this area, a deeper understanding of their denoising capabilities has been lacking. Specifically, standard linear attention mechanisms are suboptimal because they can only learn a single, average spectral denoising filter across an entire training distribution, which is insufficient for graphs exhibiting diverse spectral properties. To overcome this limitation, a new approach called Spectral Attention has been proposed, which directly incorporates the input graph's spectrum. This theoretical framework demonstrates a clear advantage over linear attention, particularly when dealing with spectrally varied graph distributions. Building on this, the researchers developed Graph Convolutional Attention (GCA), a practical and permutation-equivariant implementation. GCA achieves spectral denoising by employing graph-filtered queries and keys, allowing it to adapt more effectively to the unique spectral characteristics of each graph. Empirical evaluations confirm that GCA consistently enhances graph denoising and diffusion performance across both synthetic and real-world datasets, with improvements directly correlating with the spectral diversity of the data. Notably, GCA can match the performance of standard graph transformers without the need for computationally expensive structural feature computations. When combined with PEARL positional encodings, GCA further enables faster inference without compromising quality, making it a highly efficient and effective solution for graph-based tasks.