PyTorch Automatic Differentiation for Physics-Informed Neural Networks Explained

This research offers an in-depth, step-by-step explanation of PyTorch's automatic differentiation (AD) engine, specifically focusing on its application within Physics-Informed Neural Networks (PINNs). PINN training uniquely requires two distinct levels of differentiation: first, computing physics derivatives through the neural network itself, and second, calculating parameter gradients for a loss function that depends on these physics derivatives. The paper meticulously traces the entire computational pipeline using a simple 1-3-3-1 multilayer perceptron and an initial value problem as an example. The explanation covers the construction of the computational graph during the forward pass, followed by the reverse-mode backward traversal, which efficiently computes all parameter gradients in a single pass. A key aspect highlighted is the "graph-on-graph" mechanism enabled by `create_graph=True`, which is essential for correctly differentiating through the physics-informed residual term. Every adjoint value in the process is verified against hand derivations, establishing a clear connection between the P/Q sensitivity framework and the vector-Jacobian products utilized by PyTorch's autograd engine. This detailed breakdown provides a foundational understanding of how complex gradient computations are handled in such specialized neural network architectures.