Gradient Is All You Need? How Consensus-Based Optimization Can Be Interpreted as a Stochastic Relaxation of Gradient Descent

Abstract. In this paper, we provide a novel analytical perspective on the theoretical understanding of gradient-based learning algorithms by interpreting consensus-based optimization (CBO), a recently proposed multiparticle derivative-free optimization method, as a stochastic relaxation of gradient descent (GD). Remarkably, we observe that through communication of the particles, CBO exhibits a stochastic gradient descent (SGD)–like behavior despite solely relying on evaluations of the objective function. The fundamental value of such a link between CBO and SGD lies in the fact that CBO is provably globally convergent to global minimizers for ample classes of nonsmooth and nonconvex objective functions. Hence, on the one hand, we offer a novel explanation for the success of stochastic relaxations of GD by furnishing useful and precise insights that explain how problem-tailored stochastic perturbations of GD (like the ones induced by CBO) overcome energy barriers and reach deep levels of nonconvex functions. On the other hand, and contrary to the conventional wisdom for which derivative-free methods ought to be inefficient or not possess generalization abilities, our results unveil an intrinsic GD nature of heuristics. Instructive numerical illustrations support the theoretical insights.