A Consensus-Based Global Optimization Method with Adaptive Momentum Estimation (Communications in Computational Physics, 2022)
Objective functions in large-scale machine-learning and artificial intelligenceapplications often live in high dimensions with strong non-convexity and massivelocal minima. Gradient-based methods, such as the stochastic gradient method andAdam [15], and gradient-free methods, such as the consensus-based optimization (CBO)method, can be employed to find minima. In this work, based on the CBO method andAdam, we propose a consensus-based global optimization method with adaptive momentum estimation (Adam-CBO). Advantages of the Adam-CBO method include:• It is capable of finding global minima of non-convex objective functions withhigh success rates and low costs. This is verified by finding the global minimizerof the 1000 dimensional Rastrigin function with 100% success rate at a cost onlygrowing linearly with respect to the dimensionality.• It can handle non-differentiable activation functions and thus approximate low-regularity functions with better accuracy. This is confirmed by solving a machine learning task for partial differential equations with low-regularity solutionswhere the Adam-CBO method provides better results than Adam.• It is robust in the sense that its convergence is insensitive to the learning rate by alinear stability analysis. This is confirmed by finding the minimizer of a quadraticfunction.
Keywords: Consensus-based optimization; global optimization; machine learning; curse of dimensionality.