Overcoming the loss conditioning bottleneck in optimization-based PDE solvers: a well-conditioned loss function

Optimization-based PDE solvers that minimize scalar loss functions have gained increasing attention in recent years. These methods either define the loss directly over discrete variables, as in Optimizing a Discrete Loss (ODIL), or indirectly through a neural network surrogate, as in Physics-Informed Neural Networks (PINNs). However, despite their promise, such methods often converge much more slowly than classical iterative solvers and are commonly regarded as inefficient. We revisit the efficiency bottleneck of optimization-based PDE solvers using a classical result from numerical linear algebra: the mean squared error (MSE) loss implicitly corresponds to the normal equations, which squares the condition number and can significantly slow optimization. To address this, we propose a Stabilized Gradient Residual (SGR) loss that uses a tunable weight to interpolate between the MSE-induced normal-equation gradient and a residual-based update direction, enabling a controllable trade-off between convergence speed and training stability while recovering the MSE formulation as a limiting case. We systematically benchmark the convergence behavior and optimization stability of the SGR loss within both the ODIL framework and PINNs—employing either numerical or automatic differentiation—and compare its performance against classical iterative solvers. Numerical experiments on a range of benchmark problems demonstrate that, within the ODIL framework, the proposed SGR loss achieves orders-of-magnitude faster convergence than the MSE loss. It retains the computational advantages of explicit schemes while attaining convergence efficiencies comparable to classical implicit solvers, offering new insights for developing advanced iterative schemes. Further validation within the PINNs framework shows that, despite the high nonlinearity of neural networks, SGR has the potential to accelerate PINNs training, albeit with a narrower stability margin. These theoretical and empirical findings help bridge the performance gap between classical iterative solvers and optimization-based solvers, highlighting the central role of loss conditioning, and provide key insights for the design of more efficient PDE solvers. All code and data used in this study are available at https://github.com/Cao-WenBo/StabilizedGradientResidual .