Loss Jump During Loss Switch in Solving PDEs with Neural Networks (Communications in Computational Physics, 2024)
Using neural networks to solve partial differential equations (PDEs) is gaining popularity as an alternative approach in the scientific computing community. Neural networks can integrate different types of information into the loss function. Theseinclude observation data, governing equations, and variational forms, etc. These lossfunctions can be broadly categorized into two types: observation data loss directly constrains and measures the model output, while other loss functions indirectly model theperformance of the network, which can be classified as model loss. However, this alternative approach lacks a thorough understanding of its underlying mechanisms, including theoretical foundations and rigorous characterization of various phenomena.This work focuses on investigating how different loss functions impact the training ofneural networks for solving PDEs. We discover a stable loss-jump phenomenon: whenswitching the loss function from the data loss to the model loss, which includes different orders of derivative information, the neural network solution significantly deviatesfrom the exact solution immediately. Further experiments reveal that this phenomenonarises from the different frequency preferences of neural networks under different lossfunctions. We theoretically analyze the frequency preference of neural networks undermodel loss. This loss-jump phenomenon provides a valuable perspective for examining the underlying mechanisms of neural networks in solving PDEs.
Keywords: Loss jump; frequency bias; neural network; loss switch.