Frequency Principle: Fourier Analysis Sheds Light on Deep Neural Networks (Communications in Computational Physics, 2020)
We study the training process of Deep Neural Networks (DNNs) from theFourier analysis perspective. We demonstrate a very universal Frequency Principle(F-Principle) — DNNs often fit target functions from low to high frequencies — onhigh-dimensional benchmark datasets such as MNIST/CIFAR10 and deep neural networks such as VGG16. This F-Principle of DNNs is opposite to the behavior of Jacobimethod, a conventional iterative numerical scheme, which exhibits faster convergencefor higher frequencies for various scientific computing problems. With theories under an idealized setting, we illustrate that this F-Principle results from the smoothness/regularity of the commonly used activation functions. The F-Principle impliesan implicit bias that DNNs tend to fit training data by a low-frequency function. Thisunderstanding provides an explanation of good generalization of DNNs on most realdatasets and bad generalization of DNNs on parity function or a randomized dataset.
Keywords: Deep learning; training behavior; generalization; Jacobi iteration; Fourier analysis.