Evidence Bounds in Singular Models: Probabilistic and Variational Perspectives

In Bayesian statistics, the marginal likelihood, a.k.a. the evidence, contains an intrinsic penalty accounting for larger model sizes and is of fundamental importance in Bayesian model comparison. Over the past two decades, there has been steadily increasing activity to understand the nature of this penalty in singular statistical models, building on pioneering works by Sumio Watanabe. Unlike regular models where the Bayesian information criterion (BIC) encapsulates a first-order expansion of the evidence, parameter counting gets trickier in singular models where a quantity called the real log-canonical threshold (RLCT) summarizes the effective model dimensionality. In this article, we offer a probabilistic treatment to recover nonasymptotic versions of established evidence bounds as well as prove a new result based on the Gibbs variational inequality. In particular, we show that mean-field variational inference correctly recovers the RLCT for any singular model in its standard form. We additionally exhibit sharpness of our bound empirically in dimension d=2 and provide two conjectures concerning the asymptotics of the mean-field ELBO for singular models in standard form.