PairSAE Enhances Interpretability of Protein Foundation Models

Foundation models in structural biology have achieved impressive results in predicting biomolecular structures, showing promise for protein and small molecule design. However, understanding the internal features driving their outputs remains a significant challenge. Standard sparse autoencoders (SAEs), effective for transformer-style sequence embeddings, are not directly applicable to pairformer-like architectures due to the quadratic increase in features and the obscuring of concepts distributed across both sequence and pair representations. This research introduces PairSAE, a novel approach designed to summarize pairwise tensors using N-mode Singular Value Decomposition (SVD) into token-wise interaction roles. It then employs a sparse autoencoder to learn a shared set of token-level features that can decode into both sequence and pair representations. When evaluated on Boltz-2 activations for PLINDER protein-ligand complexes, PairSAE generated interpretable features that correlated well with UniProt annotations and accurately predicted Boltz-2 affinity values. These findings suggest that PairSAE effectively bridges the latent space of structural biology foundation models with understandable structural concepts, overcoming the limitations of conventional SAEs when dealing with pairformer-induced complexities.