New SS-ESOAP Optimizer Boosts Physics-Informed Neural Network Training Accuracy.

Physics-Informed Neural Networks (PINNs) often struggle with ill-conditioned optimization problems, which hinders their ability to achieve high accuracy. Existing methods like dense quasi-Newton approaches are computationally expensive, while Kronecker-factored methods like SOAP, though scalable, require frequent basis updates. A new optimizer, SS-ESOAP, addresses these limitations by integrating SOAP-style preconditioning with a unique scalar secant-energy correction tailored for Kronecker geometry. It also features an adaptive basis update mechanism and variance-state downscaling. This combination aims to provide a more efficient and robust solution for training PINNs. Evaluations across eight partial differential equation (PDE) benchmarks show that SS-ESOAP achieves the lowest final residual in six cases, including complex problems like Burgers and Boussinesq equations. For instance, on the Boussinesq equation, SS-ESOAP reached a residual of 10^-5 in 4.1 hours using 9.2 GB VRAM, a target Adam failed to meet within 14 hours. These results suggest SS-ESOAP is a promising scalable option for high-accuracy physics-informed training, particularly for stiff problems.