Hierarchical Attention Improves Operator Learning with Domain Decomposition

The paper introduces a new hierarchical attention mechanism, drawing inspiration from the principles of two-level overlapping Schwarz domain decomposition methods. These methods are known for their ability to combine localized corrections within subdomains with a coarser level that facilitates the communication of global, long-range information. The proposed attention mechanism applies this concept to finite-dimensional operator learning. Instead of a dense global factorization, it constructs a two-level additive structure. This involves local low-rank attention blocks operating on overlapping subdomains, which are then combined with a coarse attention block responsible for global information. Evaluations on a one-dimensional diffusion problem, a sequence-to-sequence setting where the exact nonlocal solution operator is known, demonstrated significant advantages. The domain-decomposition attention operator trained faster and yielded more accurate approximations compared to a global low-rank attention baseline, all while utilizing substantially fewer parameters. This suggests a more efficient and effective way to learn complex operators.