Expansive natural neural gradient flows for energy minimization

This paper develops expansive gradient dynamics in deep neural network (DNN)-induced mapping spaces. Specifically, we introduce a framework for minimizing a broad class of energy functionals in an abstract Hilbert space setting, with applications spanning PDE-based problems and supervised learning. The approach hinges on a Hilbert space metric in the full diffeomorphism mapping space, which can be viewed as a generalized Wasserstein-2 metric. Within this framework, we study a projected gradient descent method operating on DNN-parameterized sets. More importantly, we develop an adaptive expansion strategy to dynamically enlarge the DNN architecture. This expansion mechanism aims to enhance the alignment of the neural manifold-induced natural gradient direction, as well as possible, with the ideal Hilbert space gradient descent direction by leveraging the fact that we can evaluate projections of the Hilbert space gradient. We demonstrate the efficacy of the proposed strategy on several simple model problems involving energies arising in the context of function approximation, physics informed learning, and model reduction. Additionally, we highlight the importance of assembling the neural flow matrix based on the inner product of the ambient Hilbert space. The presented algorithms represent the simplest specifications of the broader framework, with a detailed analysis deferred to future work.