New Gradient Descent Method Boosts Neural PDE Solver Accuracy

This research presents a new optimization technique called Energy Manifold Natural Gradient Descent (EMNGD) specifically designed for neural network-based solvers for partial differential equations (PDEs). Unlike traditional gradient descent methods that assume a flat, unconstrained parameter space, EMNGD considers the intrinsic geometry of the parameter space when parameters are constrained to a Riemannian manifold. The framework refines parameter updates by aligning them with the curvature of an underlying function-space energy, ensuring that updates remain within feasible directions. This approach has been mathematically proven to provide a superior approximation to the function-space Newton vector and demonstrates robustness even with inexact solutions. Empirical evaluations on various neural PDE benchmarks show that EMNGD significantly enhances both the accuracy and speed of convergence. The method also offers scalable solutions through techniques like the Woodbury identity and Nyström approximation, allowing for efficient computation while maintaining directional accuracy.