A matrix preconditioning framework for physics-informed neural networks based on the adjoint method

Physics-informed neural networks (PINNs) have recently emerged as a popular approach for solving forward and inverse problems involving partial differential equations (PDEs). Compared to fully connected neural networks, PINNs based on convolutional neural networks (PICNNs) offer advantages in the hard enforcement of boundary conditions and in reducing the computational cost of partial derivatives. However, PICNNs still struggle with slow convergence and even failure in some scenarios. In this study, we propose a matrix preconditioning method to improve the convergence of PICNNs. Specifically, we combine automatic differentiation with matrix coloring to compute the Jacobian matrix of the PDE system, which is used to construct the preconditioner via incomplete lower-upper factorization. We subsequently use the preconditioner to scale the PDE residual in the loss function to reduce the condition number of the Jacobian matrix, which is key to improving the convergence of PICNNs. To overcome the incompatibility between automatic differentiation and triangular solves in the preconditioning, we also design a framework based on the adjoint method to compute the gradients of the loss function with respect to the network parameters. By numerical experiments, we validate that the proposed method significantly accelerates the convergence of PICNNs and successfully solves the multi-scale problem and the high Reynolds number problem, in both of which PICNNs fail to produce satisfactory results.