Unsupervised Finetuning Adapts PDE Foundation Models Without Ground Truth

Pretrained Partial Differential Equation (PDE) foundation models demonstrate strong generalization across various equations, but adapting them to new, unseen PDE systems typically demands extensive ground-truth solution data. This data is often costly to obtain or simply unavailable, posing a significant limitation. This paper addresses this challenge by proposing an unsupervised PDE-based finetuning framework.The approach begins with pretraining a neighborhood attention Transformer on a diverse range of time-dependent PDEs, covering different spatial scales to ensure transferable representations. In the subsequent adaptation stage, the model is finetuned on novel equations using a physics-based objective function derived from the PDE residual and boundary conditions, eliminating the need for ground-truth solutions.To enhance adaptation, the researchers introduce NSLoRA, a Newton-Schulz orthogonalized variant of low-rank adaptation (LoRA). NSLoRA specifically rebalances adaptation across different physical quantities, addressing uneven learning issues common in standard LoRA. The method achieves performance comparable to supervised LoRA finetuning, consistently outperforming competitive neural operator baselines and other PDE foundation models across various heterogeneous PDE benchmarks.