Unifying Dual-Encoder Networks with Low-Interaction-Rank Learning.

Researchers have developed a unified theoretical framework for multiplicative dual-encoder networks, which are widely used across various AI domains like operator learning, retrieval, and vision-language models. This framework introduces the concept of "low interaction rank," which quantifies the intrinsic complexity of functions these networks can represent through their interaction spectrum. The theory clarifies fundamental design choices, such as the number of interaction modes and normalization techniques. The paper explains that approximation error in these networks decomposes into spectral truncation and encoder-realization terms, and that sample complexity is governed by the sum of encoder complexities. It also highlights a central identifiability problem where encoders are only defined up to a linear gauge symmetry, making learned coordinates arbitrary. The framework demonstrates that normalization acts as gauge fixing, with whitening recovering true interaction modes and making concept axes interpretable, as validated by experiments on synthetic kernels, operator learning, and CLIP models.